Symbolical Reasoning about Numerical Data: A Hybrid Approach

By combining methods from artificial intelligence and signal analysis, we have developed a hybrid system for medical diagnosis. The core of the system is a fuzzy expert system with a dual source knowledge base. Two sets of rules are acquired, automatically from given examples and indirectly formulated by the physician. A fuzzy neural network serves to learn from sample data and allows to extract fuzzy rules for the knowledge base. A complex signal transformation preprocesses the digital data a priori to the symbolic representation. Results demonstrate the high accuracy of the system in the field of diagnosing electroencephalograms where it outperforms the visual diagnosis by a human expert for some phenomena.     

知识蒸馏方法研究与应用综述

随着深度学习方法的不断发展,其存储代价和计算代价也不断增长,在资源受限的平台上,这种情况给其应用带来了挑战。为了应对这种挑战,研究者提出了一系列神经网络压缩方法,其中知识蒸馏是一种简单而有效的方法,成为研究热点之一。知识蒸馏的特点在于它采用了“教师—学生”架构,使用一个大型网络指导小型网络进行训练,以提升小型网络在应用场景下的性能,从而间接达到网络压缩的目的。同时,知识蒸馏具有不改变网络结构的特性,从而具有较好的可扩展性。本文首先介绍知识蒸馏的由来以及发展,随后根据方法优化的目标将知识蒸馏的改进方法分为两大类,即面向网络性能的知识蒸馏和面向网络压缩的知识蒸馏,并对经典方法和最新方法进行系统的分析和总结,最后列举知识蒸馏方法的几种典型应用场景,以便加深对各类知识蒸馏方法原理及其应用的理解。知识蒸馏方法发展至今虽然已经取得较好的效果,但是各类知识蒸馏方法仍然有不足之处,本文也对不同知识蒸馏方法的缺陷进行了总结,并根据网络性能和网络压缩两个方面的分析,给出对知识蒸馏研究的总结和展望。     

Analysis and synthesis of feedforward neural networks usingdiscrete affine wavelet transformations

A representation of a class of feedforward neural networks in terms of discrete affine wavelet transforms is developed. It is shown that by appropriate grouping of terms, feedforward neural networks with sigmoidal activation functions can be viewed as architectures which implement affine wavelet decompositions of mappings. It is shown that the wavelet transform formalism provides a mathematical framework within which it is possible to perform both analysis and synthesis of feedforward networks. For the purpose of analysis, the wavelet formulation characterizes a class of mappings which can be implemented by feedforward networks as well as reveals an exact implementation of a given mapping in this class. Spatio-spectral localization properties of wavelets can be exploited in synthesizing a feedforward network to perform a given approximation task. Two synthesis procedures based on spatio-spectral localization that reduce the training problem to one of convex optimization are outlined.     

An architecture for a self-improving instructional planner for intelligent tutoring systems

Machine instructional planners use changing and uncertain data to incrementally configure plans and control the execution and dynamic refinement of these plans. Current instructional planners cannot adequately plan, replan, and monitor the delivery of instruction. This is due in part to the fact that current instructional planners are incapable of planning in a global context, developing competing plans in parallel, monitoring their planning behavior, and dynamically adapting their control behavior. In response to these and other deficiencies of instructional planners a generic system architecture based on the blackboard model was implemented. This self-improving instructional planner (SUP) dynamically creates instructional plans, requests execution of these plans, replans, and improves its planning behavior based on a student's responses to tutoring. Global planning was facilitated by explicitly representing decisions about past, current, and future plans on a global data structure called the plan blackboard. Planning in multiple worlds is facilitated by labeling plan decisions by the context in which they were generated. Plan monitoring was implemented as a set of monitoring knowledge sources. The flexible control capability for instructional planner was adapted from the blackboard architecture BB1. The explicit control structure of SUP enabled complex and flexible planning behavior while maintaining a simple planning architecture.     

Evolutionary Multi-task Learning for Modular Knowledge Representation in Neural Networks

The brain can be viewed as a complex modular structure with features of information processing through knowledge storage and retrieval. Modularity ensures that the knowledge is stored in a manner where any complications in certain modules do not affect the overall functionality of the brain. Although artificial neural networks have been very promising in prediction and recognition tasks, they are limited in terms of learning algorithms that can provide modularity in knowledge representation that could be helpful in using knowledge modules when needed. Multi-task learning enables learning algorithms to feature knowledge in general representation from several related tasks. There has not been much work done that incorporates multi-task learning for modular knowledge representation in neural networks. In this paper, we present multi-task learning for modular knowledge representation in neural networks via modular network topologies. In the proposed method, each task is defined by the selected regions in a network topology (module). Modular knowledge representation would be effective even if some of the neurons and connections are disrupted or removed from selected modules in the network. We demonstrate the effectiveness of the method using single hidden layer feedforward networks to learn selected n-bit parity problems of varying levels of difficulty. Furthermore, we apply the method to benchmark pattern classification problems. The simulation and experimental results, in general, show that the proposed method retains performance quality although the knowledge is represented as modules.     

A blackboard system for the off-line programming of Robots

This paper describes the use of the blackboard architecture for the off-line programming of an IMB 7565 Robot. A blackboard system was implemented in PROLOG and it has been applied successfully for the automatic generation of a control code for the robot to perform the task of block assembly in an environment with an obstacle. The opportunistic type of problem-solving offered by the blackboard architecture has succeeded in obtaining a solution. The user-interface to the system is represented as a knowledge source in the blackboard system, which allows the user to modify the goal specifications during the operation of the blackboard system.     

A comprehensive survey on optimizing deep learning models by metaheuristics

Deep neural networks (DNNs), which are extensions of artificial neural networks, can learn higher levels of feature hierarchy established by lower level features by transforming the raw feature space to another complex feature space. Although deep networks are successful in a wide range of problems in different fields, there are some issues affecting their overall performance such as selecting appropriate values for model parameters, deciding the optimal architecture and feature representation and determining optimal weight and bias values. Recently, metaheuristic algorithms have been proposed to automate these tasks. This survey gives brief information about common basic DNN architectures including convolutional neural networks, unsupervised pre-trained models, recurrent neural networks and recursive neural networks. We formulate the optimization problems in DNN design such as architecture optimization, hyper-parameter optimization, training and feature representation level optimization. The encoding schemes used in metaheuristics to represent the network architectures are categorized. The evolutionary and selection operators, and also speed-up methods are summarized, and the main approaches to validate the results of networks designed by metaheuristics are provided. Moreover, we group the studies on the metaheuristics for deep neural networks based on the problem type considered and present the datasets mostly used in the studies for the readers. We discuss about the pros and cons of utilizing metaheuristics in deep learning field and give some future directions for connecting the metaheuristics and deep learning. To the best of our knowledge, this is the most comprehensive survey about metaheuristics used in deep learning field.

     

Neural network architectures for selecting the maximum input

In this paper two neural network architectures for selecting the maximum among a set of numbers are introduced. The first architecture is recurrent and relies on the Hamming MaxNet. The second architecture is feedforward, featuring modularity and pipelineability.     

搅拌设备化工预设计专家系统的黑板系统结构

传统的手工设计混合设备的方法十分耗时且容易出错,因此实现设计的自动化和智能化是非常用意义的开发了搅拌设备化工预设计专家系统系统。详细介绍了专家系统的结构,系统采用两层黑板结构模型和多种知识源相结合的策略,来处理解决复杂的协同问题。知识源包括设计规则库、设计公式库和机械设备标准库,内部黑板主要是为规则库的推理机存储信息服务,而全局黑板系统则是整个系统的数据共享和信息交换中心,并采用层次框架结构进行表达,将人工智能技术与普通设计程序相结合,可以在设计过程为用户提供系统的帮助和指导,来实现搅拌设备设计的自动化和智能化。     

Some new neural network architectures with improved learning schemes

 Here, we present two new neuron model architectures and one modified form of existing standard feedforward architecture (MSTD). Both the new models use self-scaling scaled conjugate gradient algorithm (SSCGA) and lambda–gamma (L–G) algorithm and entail the properties of basic as well as higher order neurons (i.e., multiplication and the aggregation functions). Of these two, compensatory neural network architecture (CNNA) requires relatively smaller number of inter-neuronal connections, cuts down on the computational budget by almost 50% and speeds up convergence, besides, gives better training and prediction accuracy. The second model sigma–pi–sigma (SPS) ensures faster convergence, better training and prediction accuracy. The third model (MSTD) performs much better than the standard feedforward architecture (STD). The effect of normalizing the outputs for training also studied here shows virtually no improvement, at low iteration level, say ∼500, with increasing range of scaling. Increasing the number of neurons beyond a point also shows to have little effect in the case of higher order neuron.The numerous simulation runs for the problem of satellite orbit determination and the complex XOR problems establishes the robustness of the proposed neuron models architectures.     

An incremental neural learning framework and its application to vehicle diagnostics

This paper presents a framework for incremental neural learning (INL) that allows a base neural learning system to incrementally learn new knowledge from only new data without forgetting the existing knowledge. Upon subsequent encounters of new data examples, INL utilizes prior knowledge to direct its incremental learning. A number of critical issues are addressed including when to make the system learn new knowledge, how to learn new knowledge without forgetting existing knowledge, how to perform inference using both the existing and the newly learnt knowledge, and how to detect and deal with aged learnt systems. To validate the proposed INL framework, we use backpropagation (BP) as a base learner and a multi-layer neural network as a base intelligent system. INL has several advantages over existing incremental algorithms: it can be applied to a broad range of neural network systems beyond the BP trained neural networks; it retains the existing neural network structures and weights even during incremental learning; the neural network committees generated by INL do not interact with one another and each sees the same inputs and error signals at the same time; this limited communication makes the INL architecture attractive for parallel implementation. We have applied INL to two vehicle fault diagnostics problems: end-of-line test in auto assembly plants and onboard vehicle misfire detection. These experimental results demonstrate that the INL framework has the capability to successfully perform incremental learning from unbalanced and noisy data. In order to show the general capabilities of INL, we also applied INL to three general machine learning benchmark data sets. The INL systems showed good generalization capabilities in comparison with other well known machine learning algorithms.     

Comparison of RBF and SHL Neural Network Based Adaptive Control

Modern unmanned aerial vehicles (UAVs) are required to perform complex maneuvers while operating in increasingly uncertain environments. To meet these demands and model the system dynamics with a high degree of precision, a control system design known as neural network based model reference adaptive control (MRAC) is employed. There are currently two neural network architectures used by industry and academia as the adaptive element for MRAC; the radial basis function and single hidden layer neural network. While mathematical derivations can identify differences between the two neural networks, there have been no comparative analyses conducted on the performance characteristics for the flight controller to justify the selection of one neural network over the other. While the architecture of both neural networks contain similarities, there are several key distinctions which exhibit a noticeable impact on the control system’s overall performance. In this paper, a detailed comparison of the performance characteristics between both neural network based adaptive control approaches has been conducted in an application highly relevant to UAVs. The results and conclusions drawn from this paper will provide engineers with tangible justification for the selection of the better neural network adaptive element and thus a controller with better performance characteristics.     

一种基于强化学习的限定代价下卷积神经网结构自动化设计方法

目前的神经网络结构自动化设计方法主要对所设计神经网络结构的预测准确率进行优化。然而,实际应用中经常要求所设计的神经网络结构满足特定的代价约束,如内存占用、推断时间和训练 时间等。该文提出了一种新的限定代价下的神经网络结构自动化设计方法,选取内存占用、推断时间和训练时间三类代表性代价在 CIFAR10 数据集上进行了实验,并与现有方法进行了对比分析。该方法获得了满足特定代价约束的高准确率的卷积神经网络结构,可优化的代价种类比现有方法更多。     

A unified mathematical form for removing neurons based on orthogonal projection and crosswise propagation

It is a common practice to adjust the number of hidden neurons in training, and the removal of neurons in neural networks plays an indispensable role in this architecture manipulation. In this paper, a succinct and unified mathematical form is upgraded to the generic case for removing neurons based on orthogonal projection and crosswise propagation in a feedforward layer with different architectures of neural networks, and further developed for several neural networks with different architectures. For a trained neural network, the method is divided into three stages. In the first stage, the output vectors of the feedforward observation layer are classified to clusters. In the second stage, the orthogonal projection is performed to locate a neuron whose output vector can be approximated by the other output vectors in the same cluster with the least information loss. In the third stage, the previous located neuron is removed and the crosswise propagation is implemented in each cluster. On accomplishment of the three stages, the neural network with the pruned architecture is retrained. If the number of clusters is one, the method is degenerated into its special case with only one neuron being removed. The applications to different architectures of neural networks with an extension to the support vector machine are exemplified. The methodology supports in theory large-scale applications of neural networks in the real world. In addition, with minor modifications, the unified method is instructive in pruning other networks as far as they have similar network structure to the ones in this paper. It is concluded that the unified pruning method in this paper equips us an effective and powerful tool to simplify the architecture in neural networks.     

A New Multi-output Neural Model with Tunable Activation Function and its Applications

In this paper, a new multi-output neural model with tunable activation function (TAF) and its general form are presented. It combines both traditional neural model and TAF neural model. Recursive least squares algorithm is used to train a multilayer feedforward neural network with the new multi-output neural model with tunable activation function (MO-TAF). Simulation results show that the MO-TAF-enabled multi-layer feedforward neural network has better capability and performance than the traditional multilayer feedforward neural network and the feedforward neural network with tunable activation functions. In fact, it significantly simplifies the neural network architecture, improves its accuracy and speeds up the convergence rate.     

Preintegration lateral inhibition enhances unsupervised learning

A large and influential class of neural network architectures uses postintegration lateral inhibition as a mechanism for competition. We argue that these algorithms are computationally deficient in that they fail to generate, or learn, appropriate perceptual representations under certain circumstances. An alternative neural network architecture is presented here in which nodes compete for the right to receive inputs rather than for the right to generate outputs. This form of competition, implemented through preintegration lateral inhibition, does provide appropriate coding properties and can be used to learn such representations efficiently. Furthermore, this architecture is consistent with both neuroanatomical and neurophysiological data. We thus argue that preintegration lateral inhibition has computational advantages over conventional neural network architectures while remaining equally biologically plausible.     

INTEGRATING PLANNING,EXECUTION, AND LEARNING TO IMPROVE PLAN EXECUTION

Algorithms for planning under uncertainty require accurate action models that explicitly capture the uncertainty of the environment. Unfortunately, obtaining these models is usually complex. In environments with uncertainty, actions may produce countless outcomes and hence, specifying them and their probability is a hard task. As a consequence, when implementing agents with planning capabilities, practitioners frequently opt for architectures that interleave classical planning and execution monitoring following a replanning when failure paradigm. Though this approach is more practical, it may produce fragile plans that need continuous replanning episodes or even worse, that result in execution dead‐ends. In this paper, we propose a new architecture to relieve these shortcomings. The architecture is based on the integration of a relational learning component and the traditional planning and execution monitoring components. The new component allows the architecture to learn probabilistic rules of the success of actions from the execution of plans and to automatically upgrade the planning model with these rules. The upgraded models can be used by any classical planner that handles metric functions or, alternatively, by any probabilistic planner. This architecture proposal is designed to integrate off‐the‐shelf interchangeable planning and learning components so it can profit from the last advances in both fields without modifying the architecture.     

Learning in artificial neural systems

This paper presents an overview and analysis of teaming in artificial neural systems (ANSs). It begins with a general introduction to neural networks and connectionist approaches to information processing. The basis for learning in ANSs is then described and compared with classical machine learning. While similar in some ways, ANS learning deviates from tradition in its dependence on the modification of individual weights to bring about changes in a knowledge representation distributed across connections in a network. This unique form of learning is analyzed from two aspects: the selection of an appropriate network architecture for representing the problem, and the choice of a suitable learning rule capable of reproducing the desired function within the given network. The various network architectures are classified, and then identified with explicit restrictions on the types of functions they are capable of representing. The learning rules, i.e., algorithms that specify how the network weights are modified, are similarly taxonomized and, where possible, the limitations inherent to specific classes of rules are outlined.     

Expert networks: Paradigmatic conflict,technological rapproachement

A rule-based expert system is demonstrated to have both a symbolic computational network representation and a sub-symbolic connectionist representation. These alternate views enhance the usefulness of the original system by facilitating introduction of connectionist learning methods into the symbolic domain. The connectionist representation learns and stores metaknowledge in highly connected subnetworks and domain knowledge in a sparsely connected expert network superstructure. The total connectivity of the neural network representation approximates that of real neural systems and hence avoids scaling and memory stability problems associated with other connectionist models.Paper given to the symposiumApproaches to Cognition, the fifteenth annual Symposium in Philosophy held at the University of North Carolina, Greensboro, April 5–7, 1991.Research partially supported by the US Office of Naval Research and the Florida High Technology and Industry Council.     

Extracting reduced logic programs from artificial neural networks

Artificial neural networks can be trained to perform excellently in many application areas. Whilst they can learn from raw data to solve sophisticated recognition and analysis problems, the acquired knowledge remains hidden within the network architecture and is not readily accessible for analysis or further use: Trained networks are black boxes. Recent research efforts therefore investigate the possibility to extract symbolic knowledge from trained networks, in order to analyze, validate, and reuse the structural insights gained implicitly during the training process. In this paper, we will study how knowledge in form of propositional logic programs can be obtained in such a way that the programs are as simple as possible—where simple is being understood in some clearly defined and meaningful way.