Blind visual quality assessment for image super-resolution by convolutional neural network

Multimedia Tools and Applications - Image super-resolution aims to increase the resolution of images with good visual experience. Over the past decades, there have been many image super-resolution...     

No-reference synthetic image quality assessment with convolutional neural network and local image saliency

Depth-image-based rendering (DIBR) is widely used in 3DTV, free-viewpoint video, and interactive 3D graphics applications. Typically, synthetic images generated by DIBR-based systems incorporate various distortions, particularly geometric distortions induced by object dis-occlusion. Ensuring the quality of synthetic images is critical to maintaining adequate system service. However, traditional 2D image quality metrics are ineffective for evaluating synthetic images as they are not sensitive to geometric distortion. In this paper, we propose a novel no-reference image quality assessment method for synthetic images based on convolutional neural networks, introducing local image saliency as prediction weights. Due to the lack of existing training data, we construct a new DIBR synthetic image dataset as part of our contribution. Experiments were conducted on both the public benchmark IRCCyN/IVC DIBR image dataset and our own dataset. Results demonstrate that our proposed metric outperforms traditional 2D image quality metrics and state-of-the-art DIBR-related metrics.

     

改进深度残差卷积神经网络的LDCT图像估计

针对低剂量计算机断层扫描（Low-Dose Computed Tomography,LDCT）重建图像出现明显条形伪影的现象,提出了一种基于残差学习的深度卷积神经网络（Deep Residual Convolutional Neural Network,DR-CNN）模型,可以从LDCT图像预测标准剂量计算机断层扫描（Normal-Dose Computed Tomography,NDCT）图像。该模型在训练阶段,将数据集中的LDCT图像和NDCT图像相减得到残差图像,将LDCT图像和残差图像分别作为输入和标签,通过深度卷积神经网络（Convolution Neural Network,CNN）学习输入和标签之间的映射关系;在测试阶段,利用此映射关系从LDCT图像预测残差图像,用LDCT图像减去残差图像得到预测的NDCT图像。实验采用50对大小为512×512的同一体模的常规剂量胸腔扫描切片和投影域添加噪声后的重建图像作为数据集,其中45对作为训练集,其他作为测试集,来验证此模型的有效性。通过与非局部降噪算法、匹配三维滤波算法和K-SVD算法等目前公认效果较好的图像去噪算法对比,所提模型预测的NDCT图像均方根误差小,且信噪比略高于其他算法处理结果。     

级联优化CNN的手指静脉图像质量评估

目的 针对手动设计的手指静脉质量特征计算过程复杂、鲁棒性差、表达效果不理想等问题,提出了基于级联优化CNN（卷积神经网络）进行多特征融合的手指静脉质量评估方法。方法 以半自动化方式对手指静脉公开数据库MMCBNU_6000进行质量标注并用R-SMOTE（radom-synthetic minority over-sampling technique）算法平衡类别;将深度学习中的CNN结构应用到手指静脉质量评估并研究了不同的网络深度对表征手指静脉质量的影响;受到传统方法中将二值图像和灰度图像结合进行质量评估的启发,设计了两种融合灰度图像和二值图像的质量特征的模型：多通道CNN（MC-CNN）和级联优化CNN（CF-CNN）,MC-CNN在训练和测试时均需要同时输入二值图像和灰度图像,CF-CNN在训练时分阶段输入二值图像和灰度图像,测试时只需输入灰度图像。结果 本文设计的3种简单CNN结构（CNN-K,K=3,4,5）在MMCBNU_6000数据库上对测试集图像的分类正确率分别为93.31%、93.94%、85.63%,以灰度图像和二值图像分别作为CNN-4的输入在MMCBNU_6000数据库上对测试集图像的分类正确率对应为93.94%、91.92%,MC-CNN和CF-CNN在MMCBNU_6000数据库上对测试集图像的分类正确率分别为91.44%、94.62%,此外,与现有的其他算法相比,CF-CNN在MMCBNU_6000数据库上对高质量测试图像、低质量测试图像、整体测试集图像的分类正确率均最高。结论 实验结果表明,基于CF-CNN学习到的融合质量特征比现有的手工特征和基于单一静脉形式学习到的特征表达效果更好,可以有效地对手指静脉图像进行高、低质量的区分。     

Multi-focus image fusion with a deep convolutional neural network

As is well known, activity level measurement and fusion rule are two crucial factors in image fusion. For most existing fusion methods, either in spatial domain or in a transform domain like wavelet, the activity level measurement is essentially implemented by designing local filters to extract high-frequency details, and the calculated clarity information of different source images are then compared using some elaborately designed rules to obtain a clarity/focus map. Consequently, the focus map contains the integrated clarity information, which is of great significance to various image fusion issues, such as multi-focus image fusion, multi-modal image fusion, etc. However, in order to achieve a satisfactory fusion performance, these two tasks are usually difficult to finish. In this study, we address this problem with a deep learning approach, aiming to learn a direct mapping between source images and focus map. To this end, a deep convolutional neural network (CNN) trained by high-quality image patches and their blurred versions is adopted to encode the mapping. The main novelty of this idea is that the activity level measurement and fusion rule can be jointly generated through learning a CNN model, which overcomes the difficulty faced by the existing fusion methods. Based on the above idea, a new multi-focus image fusion method is primarily proposed in this paper. Experimental results demonstrate that the proposed method can obtain state-of-the-art fusion performance in terms of both visual quality and objective assessment. The computational speed of the proposed method using parallel computing is fast enough for practical usage. The potential of the learned CNN model for some other-type image fusion issues is also briefly exhibited in the experiments.     

无参考图像质量评价研究进展

图像质量评价一直是图像处理和计算机视觉领域的一个基础问题,图像质量评价模型也广泛应用于图像/视频编码、超分辨率重建和图像/视频视觉质量增强等相关领域。图像质量评价主要包括全参考图像质量评价、半参考图像质量评价和无参考图像质量评价。全参考图像质量评价和半参考图像质量评价分别指预测图像质量时参考信息完全可用和部分可用,而无参考图像质量评价是指预测图像质量时参考信息不可用。虽然全参考和半参考图像质量评价模型较为可靠,但在计算过程中必须依赖参考信息,使得应用场景极为受限。无参考图像质量评价模型因不需要依赖参考信息而有较强的适用性,一直都是图像质量评价领域研究的热点。本文主要概述2012—2020年国内外公开发表的无参考图像质量评价模型,根据模型训练过程中是否需要用到主观分数,将无参考图像质量评价模型分为有监督学习和无监督学习的无参考图像质量评价模型。同时,每类模型分成基于传统机器学习算法的模型和基于深度学习算法的模型。对基于传统机器学习算法的模型,重点介绍相应的特征提取策略及思想;对基于深度学习算法的模型,重点介绍设计思路。此外,本文介绍了图像质量评价在新媒体数据中的研究工作及图像质量评价的应用。最后对介绍的无参考图像质量评价模型进行总结,并指出未来可能的发展方向。     

A dyadic multi-resolution deep convolutional neural wavelet network for image classification

For almost the past four decades, image classification has gained a lot of attention in the field of pattern recognition due to its application in various fields. Given its importance, several approaches have been proposed up to now. In this paper, we will present a dyadic multi-resolution deep convolutional neural wavelets’ network approach for image classification. This approach consists of performing the classification of one class versus all the other classes of the dataset by the reconstruction of a Deep Convolutional Neural Wavelet Network (DCNWN). This network is based on the Neural Network (NN) architecture, the Fast Wavelet Transform (FWT) and the Adaboost algorithm. It consists, first, of extracting features using the FWT based on the Multi-Resolution Analysis (MRA). These features are used to calculate the inputs of the hidden layer. Second, those inputs are filtered by using the Adaboost algorithm to select the best ones corresponding to each image. Third, we create an AutoEncoder (AE) using wavelet networks of all images. Finally, we apply a pooling for each hidden layer of the wavelet network to obtain a DCNWN that permits the classification of one class and rejects all other classes of the dataset. Classification rates given by our approach show a clear improvement compared to those cited in this article.

     

A genetic programming-based convolutional neural network for image quality evaluations

Neural Computing and Applications - Monitoring the perceptual quality of digital images is fundamentally important since digital image transmissions through the Internet continue to increase...     

基于深度卷积神经网络的快速图像分类算法

为了应对大量图像的分类问题,提出一种基于深度卷积神经网络和CUDA-cuDNN并行运算的快速图像分类方法。该方法利用深度卷积神经网络自动学习特征的优势来解决手工设计特征普适性差等问题,同时结合基于CUDA架构的cuDNN并行运算策略来提高训练速度和加快分类速度,并且针对深度卷积神经网络易受参数扰动等缺点,引入批量正则化（Batch Normalization）以提高算法的鲁棒性。实验结果表明,该方法不仅大幅缩短了训练时间同时加快了图像的分类速度,而且进一步降低了图像分类的错误率。     

Wavelet enabled convolutional autoencoder based deep neural network for hyperspectral image denoising

Denoising of hyperspectral images (HSIs) is an important preprocessing step to enhance the performance of its analysis and interpretation. In reality, a remotely sensed HSI experiences disturbance from different sources and therefore gets affected by multiple noise types. However, most of the existing denoising methods concentrates in removal of a single noise type ignoring their mixed effect. Therefore, a method developed for a particular noise type doesn’t perform satisfactorily for other noise types. To address this limitation, a denoising method is proposed here, that effectively removes multiple frequently encountered noise patterns from HSI including their combinations. The proposed dual branch deep neural network based architecture works on wavelet transformed bands. The first branch of the network uses deep convolutional skip connected layers with residual learning for extracting local and global noise features. The second branch includes layered autoencoder together with subpixel upsampling that performs repeated convolution in each layer to extract prominent noise features from the image. Two hyperspectral datasets are used in the experiment to evaluate the performance of the proposed method for denoising of Gaussian, stripe and mixed noises. Experimental results demonstrate the superior performance of the proposed network compared to other state-of-the-art denoising methods with PSNR 36.74, SSIM 0.97 and overall accuracy 94.03?%.

     

红外模糊图像的无参考质量评价方法

图像质量评价是对图像处理算法的优劣给出合理的评估,在很多无法获取原始参考图像的应用场合中使用无参考质量评价方法。通过对红外图像结构分析得知图像所具有的不确定性往往是模糊性,而不是随机性,因此将模糊集理论中模糊熵的概念引入到红外图像质量评价中,提出一种针对红外模糊图像的无参考质量评价方法,并从算法的有效性、一致性和准确性三个方面进行比较分析。仿真实验结果表明,该方法具有计算复杂度低、运算速度快和主客观评价一致等特点,且在总体性能上优于均方误差(MSE)和峰值信噪比(PSNR)全参考图像质量评价方法。     

无参考模糊图像质量评价改进算法

针对传统无参考模糊图像质量评价算法存在高计算复杂度的问题,通过改进经典的二次模糊处理算法,提出一种快速有效的无参考模糊图像质量评价方法。该算法基于人眼视觉系统(HVS)特性,利用局部方差选取人眼感兴趣图像块代替整体图像,并将感兴趣图像块通过低通滤波处理,构造模糊图像块,通过计算滤波前后图像块相邻像素差值变化大小获取原始整体图像的客观质量评价参数。仿真测试结果表明,该算法与传统整体图像二次模糊算法相比,皮尔逊相关系数提高0.01,与主观评价结果更为一致;运算速度提高一倍,降低了运算复杂度。     

Quality-distinguishing and patch-comparing no-reference image quality assessment

No-reference image quality assessment (NR-IQA) based on deep learning attracts a great research attention recently. However, its performance in terms of accuracy and efficiency is still under exploring. To address these issues, in this paper, we propose a quality-distinguishing and patch-comparing NR-IQA approach based on convolutional neural network (QDPC-CNN). We improve the prediction accuracy by two proposed mechanisms: quality-distinguishing adaption and patch-comparing regression. The former trains multiple models from different subsets of a dataset and adaptively selects one for predicting quality score of a test image according to its quality level, and the latter generates patch pairs for regression under different combination strategies to make better use of reference images in network training and enlarge training data at the same time. We further improve the efficiency of network training by a new patch sampling way based on the visual importance of each patch. We conduct extensive experiments on several public databases and compare our proposed QDPC-CNN with existing state-of-the-art methods. The experimental results demonstrate that our proposed method outperforms the others both in terms of accuracy and efficiency.

     

New feature selection algorithms for no-reference image quality assessment

No reference image quality assessment (NR-IQA) is a challenging task since reference images are usually unavailable in real world scenarios. The performance of NR-IQA techniques is vastly dependent on the features utilized to predict the image quality. Many NR-IQA techniques have been proposed that extract features in different domains like spatial, discrete cosine transform and wavelet transform. These NR-IQA techniques have the possibility to contain redundant features, which result in degradation of quality score prediction. Recently impact of general purpose feature selection algorithms on NR-IQA techniques has shown promising results. But these feature selection algorithms have the tendency to select irrelevant features and discard relevant features. This paper presents fifteen new feature selection algorithms specifically designed for NR-IQA, which are based on Spearman rank ordered correlation constant (SROCC), linear correlation constant (LCC), Kendall correlation constant (KCC) and root mean squared error (RMSE). The proposed feature selection algorithms are applied on the extracted features of existing NR-IQA techniques. Support vector regression (SVR) is then applied to selected features to predict the image quality score. The fifteen newly proposed feature selection algorithms are evaluated using eight different NR-IQA techniques over three commonly used image quality assessment databases. Experimental results show that the proposed feature selection algorithms not only reduce the number of features but also improve the performance of NR-IQA techniques. Moreover, features selection algorithms based on SROCC and its combination with LCC, KCC and RMSE perform better in comparison to other proposed algorithms.     

Blind quality assessment for screen content images via convolutional neural network

With the wide propagation of cloud and mobile computing, screen content images (SCIs) have become more indispensable in our daily lives. Compared to natural scene images (NSIs), SCIs possess many particular characteristics, like mixed contents, extremely sharp edges, and text graphics. Consequently, more challenges occur in the feature extraction, which is used to reflect the distortion, during the quality assessment of SCIs. Recently, some convolutional neural network (CNN) models have been designed by automatically learning feature to evaluate the quality. In this paper, we develop a novel blind quality assessment method for SCIs via the CNN. First, compared with existing CNN-based methods, the proposed method avoids the disadvantage of training with image patches, and it is the pioneering attempt that takes the entire image as inputs. Second, instead of the image gray value, the original image is decomposed into two portions, i.e., the predicted and unpredicted portions, according to the internal generative mechanism (IGM) theory as the input of CNN. Through the CNN, all features of the image are learned automatically from beginning to end, and the network finally outputs the predicted score. Since existing SCI database is too small, to fully train the network, we collected 30000 SCIs and employed a high-accuracy full-reference quality assessment metric of SCI to compute scores as the training labels. Experimental results on SIQAD database demonstrate that the proposed method is comparable to reference-based SCI quality assessment metrics and is superior to the state-of-the-art NSI quality assessment metrics.     

Quantum convolutional neural network for image classification

Pattern Analysis and Applications - In this paper we propose two scale-inspired local feature extraction methods based on Quantum Convolutional Neural Network (QCNN) in the Tensorflow quantum...     

基于机器学习的无参考图像质量评价综述

无参考图像质量评价（NRIQA）因其广泛的应用需求一直以来都是计算机视觉及其交叉领域的研究热点。回顾近十几年来基于机器学习的典型NRIQA模型,介绍图像质量评价的常用数据库、算法性能指标、NRIQA主要难点和现有的解决方法;分析了不同模型的思想、实现、特点;最后统计对比多个数据库上的测试结果。总结研究现状、分析发展趋势,为这一领域的研究者提供文献参考。     

Pyramid-dilated deep convolutional neural network for crowd counting

Statistics on crowds in crowded scenes can reflect the density level of crowds and provide safety warnings. This is a laborious task if conducted manually. In recent years, automated crowd counting has received extensive attention in the computer vision field. However, this task is still challenging mainly due to the serious occlusion in crowds and large appearance variations caused by the viewing angles of cameras. To overcome these difficulties, a pyramid-dilated deep convolutional neural network for accurate crowd counting called PDD-CNN is proposed. PDD-CNN is based on a VGG-16 network that is designed to generate dense attribute feature maps from an image with an arbitrary size or resolution. Then, two pyramid dilated modules are adopted, each consisting of four parallel dilated convolutional layers with different rates and a parallel average pooling layer to capture the multiscale features. Finally, three cascading dilated convolutions are used to regress the density map and perform accurate count estimation. In addition, a novel training loss, combining the Euclidean loss with the structural similarity loss, is employed to attenuate the blurry effects of density map estimation. The experimental results on three datasets (ShanghaiTech, UCF_CC_50, and UCF-QNRF) demonstrate that the proposed PDD-CNN produces high-quality density maps and achieves a good counting performance.

     

Attention-based convolutional neural network for deep face recognition

Multimedia Tools and Applications - Discriminative feature embedding is of essential importance in the field of large scale face recognition. In this paper, we propose an attention-based...     

结合细粒度特征与深度卷积网络的手绘图检索

目的 传统的手绘图像检索方法主要集中在检索相同类别的图像，忽略了手绘图像的细粒度特征。对此，提出了一种新的结合细粒度特征与深度卷积网络的手绘图像检索方法，既注重通过深度跨域实现整体匹配，也实现细粒度细节匹配。方法 首先构建多通道混合卷积神经网络，对手绘图像和自然图像分别进行不同的处理；其次通过在网络中加入注意力模型来获取细粒度特征；最后将粗细特征融合，进行相似性度量，得到检索结果。结果 在不同的数据库上进行实验，与传统的尺度不变特征（SIFT）、方向梯度直方图（HOG）和深度手绘模型Deep SaN（sketch-a-net）、Deep 3DS（sketch）、Deep TSN（triplet sketch net）等5种基准方法进行比较，选取了Top-1和Top-10，在鞋子数据集上，本文方法Top-1正确率提升了12%，在椅子数据集上，本文方法Top-1正确率提升了11%，Top-10提升了3%，与传统的手绘检索方法相比，本文方法得到了更高的准确率。在实验中，本文方法通过手绘图像能在第1幅检索出绝大多数的目标图像，达到了实例级别手绘检索的目的。结论 提出了一种新的手绘图像检索方法，为手绘图像和自然图像的跨域检索提供了一种新思路，进行实例级别的手绘检索，与原有的方法相比，检索精度得到明显提升，证明了本文方法的可行性。