基于自注意力机制的亲属关系识别

目前, 基于局部注意力机制的卷积神经网络(CNNs)用于亲属关系识别特征提取获得了不错的效果, 但基于卷积神经网络的主干模型提升不明显, 同时鲜有研究者使用具有全局信息捕获能力的自注意机制. 因此, 提出使用基于一种无卷积主干特征提取网络的S-ViT模型, 即用具有自全局注意力机制的Vision Transformer作为基础主干特征提取网络, 通过构建孪生网络与具有局部注意力机制的CNN相结合, 扩大传统分类网络, 用于亲属关系识别相关问题的研究. 最终实验结果表明, 相比RFIW2020挑战赛领先的方法, 所提出的方法在亲属关系识别3个任务上获得了良好的效果, 第1个任务中获得了76.8%验证精度排名第二, 第2个和第3个任务中排名第三, 证明了该方法的可行性和有效性, 为亲属关系识别提出了一种新的解决方法.     

交叉连接的少层残差卷积神经网络

最近的研究表明,卷积神经网络的性能可以通过采用跨层连接来提高,典型的残差网络(Res Net)便通过恒等映射方法取得了非常好的图像识别效果.但是通过理论分析,在残差模块中,跨层连接线的布局并没有达到最优设置,造成信息的冗余和层数的浪费,为了进一步提高卷积神经网络的性能,文章设计了两种新型的网络结构,分别命名为C-FnetO和C-FnetT,它们在残差模块的基础上进行优化并且具有更少的卷积层层数,同时通过在MNIST,CIFAR-10,CIFAR-100和SVHN公开数据集上的一系列对比实验表明,与最先进的卷积神经网络对比,C-FnetO和C-FnetT网络获得了相对更好的图像识别效果,其中C-FnetT网络的性能最佳,在四种数据集上均取得了最高的准确率.     

自动化张量分解加速卷积神经网络

近年来,卷积神经网络（CNN）展现了强大的性能,被广泛应用到了众多领域.由于CNN参数数量庞大,且存储和计算能力需求高,其难以部署在资源受限设备上.因此,对CNN的压缩和加速成为一个迫切需要解决的问题.随着自动化机器学习（AutoML）的研究与发展,AutoML对神经网络发展产生了深远的影响.受此启发,提出了基于参数估计和基于遗传算法的两种自动化加速卷积神经网络算法.该算法能够在给定精度损失范围内自动计算出最优的CNN加速模型,有效地解决了张量分解中,人工选择秩带来的误差问题,能够有效地提升CNN的压缩和加速效果.通过在MNIST和CIFAR-10数据集上的严格测试,与原网络相比,在MNIST数据集上准确率稍微下降了0.35%,模型的运行时间获得了4.1倍的大幅提升;在CIFAR-10数据集上,准确率稍微下降了5.13%,模型的运行时间获得了0.8倍的大幅提升.     

优化搜索空间下带约束的可微分神经网络架构搜索

可微分架构搜索(DARTS)可高效、自动地设计神经网络架构,但其超网络的构建方式与派生策略的设计之间存在性能"鸿沟".针对上述问题,提出了优化搜索空间下带约束的可微分神经网络架构搜索算法.首先,以候选操作关联的架构参数为量化指标来分析超网络的训练过程,发现在派生架构中未生效的候选操作none占据了权重最大的架构参数,从...     

Deep topology network: A framework based on feedback adjustment learning rate for image classification

Convolutional Neural Network (CNN) has demonstrated its superior ability to achieve amazing accuracy in computer vision field. However, due to the limitation of network depth and computational complexity, it is still difficult to obtain the best classification results for the specific image classification tasks. In order to improve classification performance without increasing network depth, a new Deep Topology Network (DTN) framework is proposed. The key idea of DTN is based on the iteration of multiple learning rate feedback. The framework consists of multiple sub-networks and each sub-network has its own learning rate. After the determined iteration period, these learning rates can be adjusted according to the feedback of training accuracy, in the feature learning process, the optimal learning rate is updated iteratively to optimize the loss function. In practice, the proposed DTN framework is applied to several state-of-the-art deep networks, and its performance is tested by extensive experiments and comprehensive evaluations of CIFAR-10 and MNIST benchmarks. Experimental results show that most deep networks can benefit from the DTN framework with an accuracy of 99.5% on MINIST dataset, which is 5.9% higher than that on the CIFAR-10 benchmark.     

一种卷积神经网络集成的多样性度量方法

分类器模型之间的多样性是分类器集成的一个重要性能指标。目前大多数多样性度量方法都是基于基分类器模型的0/1输出结果（即Oracle 输出）进行计算,针对卷积神经网络的概率向量输出结果,仍需要将其转化为Oracle输出方式进行度量,这种方式未能充分利用卷积神经网络输出的概率向量所包含的丰富信息。针对此问题,利用多分类卷积神经网络模型的输出特性,提出了一种基于卷积神经网络的概率向量输出方式的集成多样性度量方法,建立多个不同结构的卷积神经网络基模型并在CIFAR-10和CIFAR-100数据集上进行实验。实验结果表明,与双错度量、不一致性度量和Q统计多样性度量方法相比,所提出的方法能够更好地体现模型之间的多样性,为模型选择集成提供更好的指导。     

Position Encoding Based Convolutional Neural Networks for Machine Remaining Useful Life Prediction

Accurate remaining useful life (RUL) prediction is important in industrial systems. It prevents machines from working under failure conditions, and ensures that the industrial system works reliably and efficiently. Recently, many deep learning based methods have been proposed to predict RUL. Among these methods, recurrent neural network (RNN) based approaches show a strong capability of capturing sequential information. This allows RNN based methods to perform better than convolutional neural network (CNN) based approaches on the RUL prediction task. In this paper, we question this common paradigm and argue that existing CNN based approaches are not designed according to the classic principles of CNN, which reduces their performances. Additionally, the capacity of capturing sequential information is highly affected by the receptive field of CNN, which is neglected by existing CNN based methods. To solve these problems, we propose a series of new CNNs, which show competitive results to RNN based methods. Compared with RNN, CNN processes the input signals in parallel so that the temporal sequence is not easily determined. To alleviate this issue, a position encoding scheme is developed to enhance the sequential information encoded by a CNN. Hence, our proposed position encoding based CNN called PE-Net is further improved and even performs better than RNN based methods. Extensive experiments are conducted on the C-MAPSS dataset, where our PE-Net shows state-of-the-art performance.      

融合非局部神经网络的行为检测模型

针对在视频行为检测中卷积神经网络(CNN)对时域信息理解能力不足的问题,提出了一种融合 非局部神经网络的行为检测模型。模型采用一种双分支的 CNN 结构,分别提取视频的空间特征和运动特征。 将视频单帧和视频连续帧序列作为网络输入,空间网络对视频当前帧进行 2D CNN 特征提取,时空网络采用融 合非局部模块的 3D CNN 来捕获视频帧之间的全局联系。为了进一步增强上下文语义信息,使用一种通道融合 机制来聚合双分支网络的特征,最后将融合后的特征用于帧级检测。在 UCF101-24 和 JHMDB 2 个数据集上进 行了实验,结果表明,该方法能够充分融合空间和时间维度信息,在基于视频的时空行为检测任务上具有较高 的检测精度。     

广义细胞自动机的结构及其硬件实现

该文作者曾提出了广义细胞自动机(GCA)的原理和并行算法．并且应用于网络快速包交换等动态优化问题．该文进一步讨论了这种新的广义细胞自动机的体系结构、算法的硬件实现及其电路设计。它们对于GCA的实际应用有重要意义．GCA结构不同于Hopfield神经网络(HNN)和细胞神经网络(CNN)，GCA由多层次多粒度宏细胞组成塔形结构．它具有多粒度的宏细胞动力学特征．相同粒度宏细胞之间没有交互，但不同粒度宏细胞之间存在一定程度的交互或反馈．分析和实验表明．在问题求解的优化性、实时性、硬件实现复杂性等方面．该文给出的GCA结构和硬件实现．与HNN和CNN相比有诸多优点．     

卷积神经网络在掌纹识别中的性能评估

目的 掌纹识别技术作为一种新兴的生物特征识别技术越来越受到广泛重视。深度学习是近10年来人工智能领域取得的重要突破。但是,基于深度学习的掌纹识别相关研究还比较初步,尤其缺乏深入的分析和讨论,且已有的工作使用的都是比较简单的神经网络模型。为此,本文使用多种卷积神经网络对掌纹识别进行性能评估。方法 选取比较典型的8种卷积神经网络模型,在5个掌纹数据库上针对不同网络模型、学习率、网络层数、训练数据量等进行性能评估,展开实验,并与经典的传统掌纹识别方法进行比较。结果 在不同卷积神经网络识别性能评估方面,ResNet和DenseNet超越了其他网络,并在PolyU M_B库上实现了100%的识别率。针对不同学习率、网络层数、训练数据量的实验发现,5×10^-5为比较合适的识别率;网络层数并非越深越好,VGG-16与VGG-19的识别率相当,ResNet层数由18层逐渐增加到50层,识别率则逐渐降低;参与网络训练的数据量总体来说越多越好。对比传统的非深度学习方法,卷积神经网络在识别效果方面还存在一定差距。结论 实验结果表明,对于掌纹识别,卷积神经网络也能获得较好的识别效果,但由于训练数据量不充分等原因,与传统算法的识别性能还有差距。基于卷积神经网络的掌纹识别研究还需要进一步深入开展。     

Convolutional adaptive denoising autoencoders for hierarchical feature extraction

Convolutional neural networks (CNNs) are typical structures for deep learning and are widely used in image recognition and classification. However, the random initialization strategy tends to become stuck at local plateaus or even diverge, which results in rather unstable and ineffective solutions in real applications. To address this limitation, we propose a hybrid deep learning CNN-AdapDAE model, which applies the features learned by the AdapDAE algorithm to initialize CNN filters and then train the improved CNN for classification tasks. In this model, AdapDAE is proposed as a CNN pre-training procedure, which adaptively obtains the noise level based on the principle of annealing, by starting with a high level of noise and lowering it as the training progresses. Thus, the features learned by AdapDAE include a combination of features at different levels of granularity. Extensive experimental results on STL-10, CIFAR-10, andMNIST datasets demonstrate that the proposed algorithm performs favorably compared to CNN (random filters), CNNAE (pre-training filters by autoencoder), and a few other unsupervised feature learning methods.     

Interpretable Relative Squeezing bottleneck design for compact convolutional neural networks model

Convolutional neural networks (CNN) are mainly used for image recognition tasks. However, some huge models are infeasible for mobile devices because of limited computing and memory resources. In this paper, feature maps of DenseNet and CondenseNet are visualized. It could be observed that there are some feature channels in locked state and some have similar distribution property, which could be compressed further. Thus, in this work, a novel architecture — RSNet is introduced to improve the computing efficiency of CNNs. This paper proposes Relative-Squeezing (RS) bottleneck design, where the output is the weighted percentage of input channels. Besides, RSNet also contains multiple compression layers and learned group convolutions (LGCs). By eliminating superfluous feature maps, relative squeezing and compression layers only transmit the most significant features to the next layer. Less parameters are employed and much computation is saved. The proposed model is evaluated on three benchmark datasets: CIFAR-10, CIFAR-100 and ImageNet. Experiment results show that RSNet performs better with less parameters and FLOPs, compared to the state-of-the-art baseline, including CondenseNet, MobileNet and ShuffleNet.     

Compressing convolutional neural networks with cheap convolutions and online distillation

Visual impairment assistance systems play a vital role in improving the standard of living for visually impaired people (VIP). With the development of deep learning technologies and assistive devices, many assistive technologies for VIP have achieved remarkable success in environmental perception and navigation. In particular, convolutional neural network (CNN)-based models have surpassed the level of human recognition and achieved a strong generalization ability. However, the large memory and computation consumption in CNNs have been one of the main barriers to deploying them into resource-limited systems for visual impairment assistance applications. To this end, most cheap convolutions (e.g., group convolution, depth-wise convolution, and shift convolution) have recently been used for memory and computation reduction but with a specific architecture design. Furthermore, it results in a low discriminability of the compressed networks by directly replacing the standard convolution with these cheap ones. In this paper, we propose to use knowledge distillation to improve the performance of compact student networks with cheap convolutions. In our case, the teacher is a network with the standard convolution, while the student is a simple transformation of the teacher architecture without complicated redesigning. In particular, we introduce a novel online distillation method, which online constructs the teacher network without pre-training and conducts mutual learning between the teacher and student network, to improve the performance of the student model. Extensive experiments demonstrate that the proposed approach achieves superior performance to simultaneously reduce memory and computation overhead of cutting-edge CNNs on different datasets, including CIFAR-10/100 and ImageNet ILSVRC 2012, compared to the previous CNN compression and acceleration methods. The codes are publicly available at https://github.com/EthanZhangYC/OD-cheap-convolution.     

卷积重提取特征的文档列表排序学习方法

在许多信息检索任务中,为了进一步提高检索性能,通常需要对检索到的文档进行重新排序,目前的排序学习方法主要集中在损失函数的构造上,而没有考虑特征之间的关系。该文将多通道深度卷积神经网络作用于文档列表排序学习方法,即ListCNN,实现了信息检索的精确重排序。由于从文档中提取的多个特征中有一些特征具有局部相关性和冗余性,因此,文中使用卷积神经网络来重新提取特征,以提高列表方法的性能。ListCNN架构考虑了原始文档特征的局部相关性,能够有效地重新提取代表性特征。在公共数据集LETOR 4.0上对ListCNN进行实验验证,结果表明其性能优于已有文档列表方法。     

VLSI implementation of transcendental function hyperbolic tangent for deep neural network accelerators

Extensive use of neural network applications prompted researchers to customize a design to speed up their computation based on ASIC implementation. The choice of activation function (AF) in a neural network is an essential requirement. Accurate design architecture of an AF in a digital network faces various challenges as these AF require more hardware resources because of its non-linear nature. This paper proposed an efficient approximation scheme for hyperbolic tangent (tanh) function which purely based on combinational design architecture. The approximation is based on mathematical analysis by considering maximum allowable error in a neural network. The results prove that the proposed combinational design of an AF is efficient in terms of area, power and delay with negligible accuracy loss on MNIST and CIFAR-10 benchmark datasets. Post synthesis results show that the proposed design area is reduced by 66% and delay is reduced by nearly 16% compared to state-of-the-art.     

一种高效的稀疏卷积神经网络加速器的设计与实现

针对卷积神经网络计算硬件化实现困难的问题,之前大部分卷积神经网络加速器的设计都集中于解决计算性能和带宽瓶颈,忽视了卷积神经网络稀疏性对加速器设计的重要意义,近来少量的能够利用稀疏性的卷积神经网络加速器设计也往往难以同时兼顾计算灵活度、并行效率和资源开销。本文首先比较了不同并行展开方式对利用稀疏性的影响,分析了利用稀疏性的不同方法,然后提出了一种能够利用激活稀疏性加速卷积神经网络计算的同时,相比于同领域其他设计,并行效率更高、额外资源开销更小的并行展开方法,最后完成了这种卷积神经网络加速器的设计并在FPGA上实现。研究结果表明:运行VGG-16网络,在ImageNet数据集下,该并行展开方法实现的稀疏卷积神经网络加速器和使用相同器件的稠密网络设计相比,卷积性能提升了108.8%,整体性能提升了164.6%,具有明显的性能优势。     

面向卷积神经网络的FPGA加速器架构设计

随着人工智能的快速发展,卷积神经网络(CNN)在很多领域发挥着越来越重要的作用。分析研究了现有卷积神经网络模型,设计了一种基于现场可编程门阵列(FPGA)的卷积神经网络加速器。在卷积运算中四个维度方向实现了并行化计算;提出了参数化架构设计,在三种参数条件下,单个时钟周期分别能够完成512、1024、2048次乘累加;设计了片内双缓存结构,减少片外存储访问的同时实现了有效的数据复用;使用流水线实现了完整的神经网络单层运算过程,提升了运算效率。与CPU、GPU以及相关FPGA加速方案进行了对比实验,实验结果表明,所提出的设计的计算速度达到了560.2 GOP/s,为i7-6850K CPU的8.9倍。同时,其计算的性能功耗比达到了NVDIA GTX 1080Ti GPU的3.0倍,与相关研究相比,所设计的加速器在主流CNN网络的计算上实现了较高的性能功耗比,同时不乏通用性。     

CoNNa–Hardware accelerator for compressed convolutional neural networks

In this paper, we propose a novel Convolutional Neural Network hardware accelerator called CoNNA, capable of accelerating pruned, quantized CNNs. In contrast to most existing solutions, CoNNA offers a complete solution to the compressed CNN acceleration, being able to accelerate all layer types commonly found in contemporary CNNs. CoNNA is designed as a coarse-grained reconfigurable architecture, which uses rapid, dynamic reconfiguration during CNN layer processing. The CoNNA architecture enables the on-the-fly selection of the CNN network that should be accelerated and also supports the acceleration of CNN networks with dynamic topology. Furthermore, by being able to directly process compressed feature and kernel maps, and skip all ineffectual computations during CNN layer processing, the CoNNA CNN accelerator is able to achieve higher CNN processing rates than some of the previously proposed solutions. The CoNNA architecture has been implemented using Xilinx ZynqUtrascale+ FPGA family and compared with seven previously proposed CNN hardware accelerators. Results of the experiments seem to indicate that the CoNNA architecture is up to 14.10, 6.05, 4.91, 2.67, 11.30, 3.08 and 3.58 times faster than previously proposed MIT's Eyeriss, NullHop, NVIDIA's Deep Learning Accelerator (NVDLA), NEURAghe, CNN_A1, fpgaConvNet, and Deephi's Aristotle CNN accelerators respectively, while using identical number of computing units and operating at the same clock frequency.     

利用自适应假近邻方法优化卷积神经网络的图像目标分类算法

为了能够在处理不同的数据类型或任务时得到良好的结果,设计了基于自适应假近邻方法的卷积神经网络（CNN）架构。将中心矩的思想应用在CNN的池化操作中,利用稀疏滤波算法实现训练过程的无监督化,并设置CNN算法的卷积掩模（卷积核）的大小和每层卷积单位（CNN神经元）的数量;此外,该架构还利用自适应假近邻方法实现了简化建模和预测等任务。实验结果证实,提出的改进CNN架构的复杂度较低,它可以更快地接受训练并且不易产生过度拟合。     

Combined neural network model to compute wavelet coefficients

Abstract: In recent years a novel model based on artificial neural networks technology has been introduced in the signal processing community for modelling the signals under study. The wavelet coefficients characterize the behaviour of the signal and computation of the wavelet coefficients is particularly important for recognition and diagnostic purposes. Therefore, we dealt with wavelet decomposition of time-varying biomedical signals. In the present study, we propose a new approach that takes advantage of combined neural network (CNN) models to compute the wavelet coefficients. The computation was provided and expressed by applying the CNNs to ophthalmic arterial and internal carotid arterial Doppler signals. The results were consistent with theoretical analysis and showed good promise for discrete wavelet transform of the time-varying biomedical signals. Since the proposed CNNs have high performance and require no complicated mathematical functions of the discrete wavelet transform, they were found to be effective for the computation of wavelet coefficients.