Hybridization of autoregressive integrated moving average (ARIMA) with probabilistic neural networks (PNNs)

Autoregressive integrated moving average (ARIMA) models are one of the most important time series models applied in financial market forecasting over the past three decades. Improving forecasting especially time series forecasting accuracy is an important yet often difficult task facing forecasters. Both theoretical and empirical findings have indicated that integration of different models can be an effective way of improving upon their predictive performance, especially when the models in the ensemble are quite different. In the literature, several hybrid techniques have been proposed by combining different time series models together, in order to yield results that are more accurate. In this paper, a new hybrid model of the autoregressive integrated moving average (ARIMA) and probabilistic neural network (PNN), is proposed in order to yield more accurate results than traditional ARIMA models. In proposed model, the estimated values of the ARIMA model are modified based on the distinguished trend of the ARIMA residuals and optimum step length, which are respectively obtained from a probabilistic neural network and a mathematical programming model. Empirical results with three well-known real data sets indicate that the proposed model can be an effective way in order to construct a more accurate hybrid model than ARIMA model. Therefore, it can be used as an appropriate alternative model for forecasting tasks, especially when higher forecasting accuracy is needed.     

深度神经网络结构搜索综述

深度神经网络在图像识别、语言识别和机器翻译等人工智能任务中取得了巨大进展,很大程度上归功于优秀的神经网络结构设计。神经网络大都由手工设计,需要专业的机器学习知识以及大量的试错。为此,自动化的神经网络结构搜索成为研究热点。神经网络结构搜索（neural architecture search,NAS）主要由搜索空间、搜索策略与性能评估方法3部分组成。在搜索空间设计上,出于计算量的考虑,通常不会搜索整个网络结构,而是先将网络分成几块,然后搜索块中的结构。根据实际情况的不同,可以共享不同块中的结构,也可以对每个块单独搜索不同的结构。在搜索策略上,主流的优化方法包含强化学习、进化算法、贝叶斯优化和基于梯度的优化等。在性能评估上,为了节省计算时间,通常不会将每一个网络都充分训练到收敛,而是通过权值共享、早停等方法尽可能减小单个网络的训练时间。与手工设计的网络相比,神经网络结构搜索得到的深度神经网络具有更好的性能。在ImageNet分类任务上,与手工设计的MobileNetV2相比,通过神经网络结构搜索得到的MobileNetV3减少了近30%的计算量,并且top-1分类精度提升了3.2%;在Cityscapes语义分割任务上,与手工设计的DeepLabv3+相比,通过神经网络结构搜索得到的Auto-DeepLab-L可以在没有ImageNet预训练的情况下,达到比DeepLabv3+更高的平均交并比（mean intersection over union,mIOU）,同时减小一半以上的计算量。神经网络结构搜索得到的深度神经网络通常比手工设计的神经网络有着更好的表现,是未来神经网络设计的发展趋势。     

Optimization of material removal rate in wire EDM by polynomial neural network models

Maximization of material removal rate may yield impractical results if the cost for generation of removal rate is not included in the optimization. Presently, there are many studies which have applied different linear, nonlinear, and metaheuristics optimization techniques to maximize material removal rate but none of the approaches have the capability of self-adaptation. In none of these studies impact of cost on removal rate was considered. That is why, in the present study, the Group Method of Data Handling which is a technique to develop polynomial neural network (PNN) models was utilized for the first time to maximize material removal rate with the help of the on- and off-stage current as the design variables constrained by the operating costs. The PNN architecture is widely used in many fields of technology and science. However, application of this architecture is scarce in case of optimization of material removal rate. According to the results the accuracy level and time of convergence in case of PNN was found to be better compared to that from artificial neural network and linear models both of which were utilized for estimation of the output variable.     

深度神经架构搜索综述

深度学习在图像、语音、文本等多种模态的数据任务上取得了优异的效果然而,针对特定任务,人工设计网络需要花费大量的时间,并且需要设计者具有一定水平的专业知识和设计经验.面对如今日趋复杂的网络架构,仅依靠人工进行设计变得越来越复杂基于此,借助算法自动地对神经网络进行架构的搜索成为了研究热点神经架构搜索的方法涉及3个方面:搜索...     

基于FPGA模拟的阻变神经网络加速器评估方法

基于阻变器件的存算一体神经网络加速器需在架构设计初期进行仿真评估,确保神经网络精度符合设计要求,但传统阻变神经网络加速器的软件模拟器运行速度较慢,难以应对大规模网络的架构评估需求。为加快仿真评估速度,设计一种基于现场可编程门阵列（FPGA）模拟的阻变神经网络加速器评估方法,分析现有阻变神经网络加速器的架构通用性,利用FPGA资源的高度并行性和运行时指令驱动的灵活模拟方式,通过硬件资源的分时复用实现多层次存算一体架构和指令集的功能模拟及主流神经网络的快速性能评估。实验结果表明,针对不同规模的忆阻器阵列和深度神经网络,该评估方法相比MNSIM和DNN NeuroSim软件模拟器运行速度分别提升了40.0~252.9倍和194.7~234.2倍。     

优化搜索空间下带约束的可微分神经网络架构搜索

可微分架构搜索(DARTS)可高效、自动地设计神经网络架构,但其超网络的构建方式与派生策略的设计之间存在性能"鸿沟".针对上述问题,提出了优化搜索空间下带约束的可微分神经网络架构搜索算法.首先,以候选操作关联的架构参数为量化指标来分析超网络的训练过程,发现在派生架构中未生效的候选操作none占据了权重最大的架构参数,从...     

基于Matlab的概率神经网络的实现及应用

介绍概率神经网络(PNN)的模型和基本算法,以及利用M atlab神经网络工具箱设计PNN网络的方法和步骤,实现对网络的设计、训练、仿真。针对某水泥泵车臂架细部焊接结构的疲劳寿命实验数据,应用PNN的分类功能对实验数据进行训练仿真处理,得到对臂架细部结构疲劳寿命的预测分类结果,验证了此方法的可靠性。     

Implementing heuristics as an optimization criterion in neural networks for job-shop scheduling

Recently neural network architectures have been developed that are capable of solving deterministic job-shop scheduling problems, part of the large class of NP-complete problems. In these architectures, however, no valid optimization criterion has been implemented. In this paper an enhanced neural network architecture for job-shop scheduling is proposed in which general rules of thumb for job-shop scheduling have been incorporated as a local optimization criterion. Implementation of the rules of thumb, by adaptation of the network architecture, results in a network that actually incorporates the optimization criterion, enabling parallel hardware implementation. Owing to the implemented local optimization criterion the performance of the network architecture is superior to previously presented architectures. Comparison with advanced heuristic sequential schedulers showed equal performance with respect to the quality of the solutions and better performance with respect to calculation speed.     

Automated design of error-resilient and hardware-efficient deep neural networks

Applying deep neural networks (DNNs) in mobile and safety-critical systems, such as autonomous vehicles, demands a reliable and efficient execution on hardware. The design of the neural architecture has a large influence on the achievable efficiency and bit error resilience of the network on hardware. Since there are numerous design choices for the architecture of DNNs, with partially opposing effects on the preferred characteristics (such as small error rates at low latency), multi-objective optimization strategies are necessary. In this paper, we develop an evolutionary optimization technique for the automated design of hardware-optimized DNN architectures. For this purpose, we derive a set of inexpensively computable objective functions, which enable the fast evaluation of DNN architectures with respect to their hardware efficiency and error resilience. We observe a strong correlation between predicted error resilience and actual measurements obtained from fault injection simulations. Furthermore, we analyze two different quantization schemes for efficient DNN computation and find one providing a significantly higher error resilience compared to the other. Finally, a comparison of the architectures provided by our algorithm with the popular MobileNetV2 and NASNet-A models reveals an up to seven times improved bit error resilience of our models. We are the first to combine error resilience, efficiency, and performance optimization in a neural architecture search framework.

     

事件检测支持向量机模型与神经网络模型比较

针对交通领域中的事件检测(无事件模式和有事件模式)模式识别问题,描述了支持向量机(SVM)的基本方法,建立了基于线性(linearfunction)、多项式(polynomialfunction)和径向基(radialbasisfunction)3种核函数的事件检测SVM模型,并与PNN、MLF模型进行了理论比较。采用I-880线圈数据集和事件数据集建立并验证SVM、PNN和MLF模型,结果发现:无论对于向北、向南或混合方向的事件检测,SVM模型的检测率(DR)、误报率(FAR)和平均检测时间(MTTD)指标均比MLF模型好;PNN模型的DR比SVM(P)模型的高,但FAR和MTTD指标不比SVM(P)模型好;在3个SVM模型中,SVM(P)检测效果最好,SVM(L)最差。SVM算法与神经网络算法相比具有避免局部最小,实现全局最优化,更好的泛化效果的优点,是高速公路事件检测的一种很有潜力的算法。     

一种基于强化学习的限定代价下卷积神经网结构自动化设计方法

目前的神经网络结构自动化设计方法主要对所设计神经网络结构的预测准确率进行优化。然而,实际应用中经常要求所设计的神经网络结构满足特定的代价约束,如内存占用、推断时间和训练 时间等。该文提出了一种新的限定代价下的神经网络结构自动化设计方法,选取内存占用、推断时间和训练时间三类代表性代价在 CIFAR10 数据集上进行了实验,并与现有方法进行了对比分析。该方法获得了满足特定代价约束的高准确率的卷积神经网络结构,可优化的代价种类比现有方法更多。     

深度学习在多核缓存预取中的应用研究综述

当前人工智能技术应用于系统结构领域的研究前景广阔,特别是将深度学习应用于多核架构的数据预取研究已经成为国内外的研究热点。针对基于深度学习的缓存预取任务进行了研究,形式化地定义了深度学习缓存预取模型。在介绍当前常见的多核缓存架构和预取技术的基础上,全面分析了现有基于深度学习的典型缓存预取器的设计思路。深度学习神经网络在多核缓存预取领域的应用主要采用了深度神经网络、循环神经网络、长短期记忆网络和注意力机制等机器学习方法,综合对比分析现有基于深度学习的数据预取神经网络模型后发现,基于深度学习的多核缓存预取技术在计算成本、模型优化和实用性等方面还存在着局限性,未来在自适应预取模型以及神经网络预取模型的实用性方面还有很大的研究探索空间和发展前景。     

SRNET: A Shallow Skip Connection Based Convolutional Neural Network Design for Resolving Singularities

Convolutional neural networks (CNNs) have shown tremendous progress and performance in recent years. Since emergence, CNNs have exhibited excellent performance in most of classification and segmentation tasks. Currently, the CNN family includes various architectures that dominate major vision-based recognition tasks. However, building a neural network (NN) by simply stacking convolution blocks inevitably limits its optimization ability and introduces overfitting and vanishing gradient problems. One of the key reasons for the aforementioned issues is network singularities, which have lately caused degenerating manifolds in the loss landscape. This situation leads to a slow learning process and lower performance. In this scenario, the skip connections turned out to be an essential unit of the CNN design to mitigate network singularities. The proposed idea of this research is to introduce skip connections in NN architecture to augment the information flow, mitigate singularities and improve performance. This research experimented with different levels of skip connections and proposed the placement strategy of these links for any CNN. To prove the proposed hypothesis, we designed an experimental CNN architecture, named as Shallow Wide ResNet or SRNet, as it uses wide residual network as a base network design. We have performed numerous experiments to assess the validity of the proposed idea. CIFAR-10 and CIFAR-100, two well-known datasets are used for training and testing CNNs. The final empirical results have shown a great many of promising outcomes in terms of performance, efficiency and reduction in network singularities issues.

     

Probabilistic neural networks and the polynomial Adaline ascomplementary techniques for classification

Two methods for classification based on the Bayes strategy and nonparametric estimators for probability density functions are reviewed. The two methods are named the probabilistic neural network (PNN) and the polynomial Adaline. Both methods involve one-pass learning algorithms that can be implemented directly in parallel neural network architectures. The performances of the two methods are compared with multipass backpropagation networks, and relative advantages and disadvantages are discussed. PNN and the polynomial Adaline are complementary techniques because they implement the same decision boundaries but have different advantages for applications. PNN is easy to use and is extremely fast for moderate-sized databases. For very large databases and for mature applications in which classification speed is more important than training speed, the polynomial equivalent can be found.     

Neural-network-based blackboard demon subsystems

We propose two neural network architectures involving feedforward and probabilistic neural network models to simulate the blackboard demon subsystem, which is responsible for triggering knowledge sources in accordance with the complex monitoring conditions on the hypotheses in the blackboard. Both architectures involve a preprocessor module to perform the transformation between symbolic hypotheses and numerical ones. They learn and record the triggering relationships between hypotheses and knowledge sources in the network links and perform better blackboard monitoring function than the traditional symbolic demon subsystems. In comparison, the probabilistic neural network-like architecture performs better when there is a possibility of using a centralized knowledge representation and when it only involves one-to-one or one-to-many mapping between hypotheses and triggering patterns. The feedforward architecture may be useful when a distributed knowledge representation is possible and when time requirements for training the architecture to learn the complex mappings are not too strict.     

Evolutionary design of neural network architectures: a review of three decades of research

We present a comprehensive review of the evolutionary design of neural network architectures. This work is motivated by the fact that the success of an Artificial Neural Network (ANN) highly depends on its architecture and among many approaches Evolutionary Computation, which is a set of global-search methods inspired by biological evolution has been proved to be an efficient approach for optimizing neural network structures. Initial attempts for automating architecture design by applying evolutionary approaches start in the late 1980s and have attracted significant interest until today. In this context, we examined the historical progress and analyzed all relevant scientific papers with a special emphasis on how evolutionary computation techniques were adopted and various encoding strategies proposed. We summarized key aspects of methodology, discussed common challenges, and investigated the works in chronological order by dividing the entire timeframe into three periods. The first period covers early works focusing on the optimization of simple ANN architectures with a variety of solutions proposed on chromosome representation. In the second period, the rise of more powerful methods and hybrid approaches were surveyed. In parallel with the recent advances, the last period covers the Deep Learning Era, in which research direction is shifted towards configuring advanced models of deep neural networks. Finally, we propose open problems for future research in the field of neural architecture search and provide insights for fully automated machine learning. Our aim is to provide a complete reference of works in this subject and guide researchers towards promising directions.

     

关系结构中的神经计算

人工神经网络在许多应用领域中建立了一种新的高度并行计算的结构。具有表示概念和知识的模糊集合被认为是处理人们日常生活中不确定问题的一种工具。特别是关系结构,它在构造现实世界中的形式关系模型中起着非常重要的作用。本文根据模糊集合理论,提出新的神经网络结构及其新的学习算法,并研究其性质。一方面把模糊集合的方法学应用于神经网络结构和学习算法的研究,同时使得神经网络的硬件实现更加容易。     

Multi-label incremental learning applied to web page categorization

Multi-label problems are challenging because each instance may be associated with an unknown number of categories, and the relationship among the categories is not always known. A large amount of data is necessary to infer the required information regarding the categories, but these data are normally available only in small batches and distributed over a period of time. In this work, multi-label problems are tackled using an incremental neural network known as the evolving Probabilistic Neural Network (ePNN). This neural network is capable of continuous learning while maintaining a reduced architecture, so that it can always receive training data when available with no drastic growth of its structure. We carried out a series of experiments on web page data sets and compared the performance of ePNN to that of other multi-label categorizers. On average, ePNN outperformed the other categorizers in four out of five metrics used for evaluation, and the structure of ePNN was less complex than that of the other algorithms evaluated.     

DeepMaker: A multi-objective optimization framework for deep neural networks in embedded systems

Deep Neural Networks (DNNs) are compute-intensive learning models with growing applicability in a wide range of domains. Due to their computational complexity, DNNs benefit from implementations that utilize custom hardware accelerators to meet performance and response time as well as classification accuracy constraints. In this paper, we propose DeepMaker framework that aims to automatically design a set of highly robust DNN architectures for embedded devices as the closest processing unit to the sensors. DeepMaker explores and prunes the design space to find improved neural architectures. Our proposed framework takes advantage of a multi-objective evolutionary approach that exploits a pruned design space inspired by a dense architecture. DeepMaker considers the accuracy along with the network size factor as two objectives to build a highly optimized network fitting with limited computational resource budgets while delivers an acceptable accuracy level. In comparison with the best result on the CIFAR-10 dataset, a generated network by DeepMaker presents up to a 26.4x compression rate while loses only 4% accuracy. Besides, DeepMaker maps the generated CNN on the programmable commodity devices, including ARM Processor, High-Performance CPU, GPU, and FPGA.     

A unified mathematical form for removing neurons based on orthogonal projection and crosswise propagation

It is a common practice to adjust the number of hidden neurons in training, and the removal of neurons in neural networks plays an indispensable role in this architecture manipulation. In this paper, a succinct and unified mathematical form is upgraded to the generic case for removing neurons based on orthogonal projection and crosswise propagation in a feedforward layer with different architectures of neural networks, and further developed for several neural networks with different architectures. For a trained neural network, the method is divided into three stages. In the first stage, the output vectors of the feedforward observation layer are classified to clusters. In the second stage, the orthogonal projection is performed to locate a neuron whose output vector can be approximated by the other output vectors in the same cluster with the least information loss. In the third stage, the previous located neuron is removed and the crosswise propagation is implemented in each cluster. On accomplishment of the three stages, the neural network with the pruned architecture is retrained. If the number of clusters is one, the method is degenerated into its special case with only one neuron being removed. The applications to different architectures of neural networks with an extension to the support vector machine are exemplified. The methodology supports in theory large-scale applications of neural networks in the real world. In addition, with minor modifications, the unified method is instructive in pruning other networks as far as they have similar network structure to the ones in this paper. It is concluded that the unified pruning method in this paper equips us an effective and powerful tool to simplify the architecture in neural networks.