Computational versus psychophysical bottom-up image saliency: a comparative evaluation study

The predictions of 13 computational bottom-up saliency models and a newly introduced Multiscale Contrast Conspicuity(MCC) metric are compared with human visual conspicuity measurements. The agreement between human visual conspicuity estimates and model saliency predictions is quantified through their rank order correlation. The maximum of the computational saliency value over the target support area correlates most strongly with visual conspicuity for 12 of the 13 models. A simple multiscale contrast model and the MCC metric both yield the largest correlation with human visual target conspicuity (>0:84). Local image saliency largely determines human visual inspection and interpretation of static and dynamic scenes. Computational saliency models therefore have a wide range of important applications, like adaptive content delivery, region-of-interest-based image compression, video summarization, progressive image transmission, image segmentation, image quality assessment, object recognition, and content-aware image scaling. However, current bottom-up saliency models do not incorporate important visual effects like crowding and lateral interaction. Additional knowledge about the exact nature of the interactions between the mechanisms mediating human visual saliency is required to develop these models further. The MCC metric and its associated psychophysical saliency measurement procedure are useful tools to systematically investigate the relative contribution of different feature dimensions to overall visual target saliency.     

基于视觉注意力变化的网络丢包视频质量评估

针对网络中受丢包损伤的视频提出了一种基于视觉注意力变化的全参考客观质量评估方法.该方法基于视觉显著性检测在视频数据上的应用,考察受网络丢包失真影响的视频数据与标准参考数据在空间和时间上引起的视觉注意力变化,并根据此变化相应的视觉显著性在空间和时间上的差异,提出了一组客观质量评估方法.文中采用17个受丢包损伤的视频数据进行测试,并实施了主观评价实验作为评价标准.与传统的没有考虑人眼视觉显著特性的质量评估方法,以及目前主流的基于视觉显著区域/感兴趣区域对失真像素进行加权的方法进行对比,实验结果表明, 基于视觉注意力变化的方法较后两者与主观质量评估结果有更好的相关性, 能够更有效地评估丢包损伤视频的质量.     

尺度自适应的SAR图像显著性检测方法

人类视觉系统能够通过对场景中感兴趣的不同事物进行显著性检测,有效地配置处理资源。基于视觉注意机制的显著性检测方法能够简化遥感影像场景分析、目标解译的复杂程度,节省处理资源。以视觉注意机制为基础,提出了一种尺度自适应的SAR图像显著性检测方法,通过不同尺度下的局部复杂度和自差异性来度量图像的显著性测度,设计显著性尺度确定算法以及融合显著性尺度和显著性测度以生成显著图,完成显著性检测的流程。实验结果表明该方法能够有效应用于SAR图像显著性检测,较之其他主流显著区域检测算法更适用于SAR图像场景分析。     

Coder selection for lossy compression of still images

A criterion for selecting a lossy coder for still images is proposed. The “optimum” coder using the proposed criterion is selected to minimize the maximum Chernoff information between the distribution for the original and any distribution from the optimal achievable region. The resulting coder has the property that the best achievable probability of error over the problems of hypothesis testing between these distributions, is greater than the best Bayesian error probability for any other choice of coder. The coder selection procedure may be applied without a knowledge of what distortion measure is more suitable for images or a knowledge of the properties of the human visual perception. Several examples of coder selection are included to illustrate the procedure.     

Familiarity based unified visual attention model for fast and robust object recognition

Even though visual attention models using bottom-up saliency can speed up object recognition by predicting object locations, in the presence of multiple salient objects, saliency alone cannot discern target objects from the clutter in a scene. Using a metric named familiarity, we propose a top-down method for guiding attention towards target objects, in addition to bottom-up saliency. To demonstrate the effectiveness of familiarity, the unified visual attention model (UVAM) which combines top-down familiarity and bottom-up saliency is applied to SIFT based object recognition. The UVAM is tested on 3600 artificially generated images containing COIL-100 objects with varying amounts of clutter, and on 126 images of real scenes. The recognition times are reduced by 2.7× and 2×, respectively, with no reduction in recognition accuracy, demonstrating the effectiveness and robustness of the familiarity based UVAM.     

Meta-metric for saliency detection evaluation metrics based on application preference

Existing saliency detection evaluation metrics often produce inconsistent evaluation results. Because of the widespread application of image saliency detection, we propose a meta-metric to evaluate the performance of these metrics based on the preference of an application that uses saliency maps as weighting maps. This study uses content-based image retrieval (CBIR) as the representative application. First, we perform CBIR using image features extracted from deep convolutional layers of convolutional neural networks as well as saliency maps computed by various saliency detection algorithms as the weighting maps over queries. Second, we establish the preference order of the saliency detection algorithms in the CBIR application by sorting the mean average precision. Third, we determine the preference order of these algorithms using existing saliency detection evaluation metrics. Finally, our meta-metric evaluates these metrics by correlating the preference order in the CBIR application with that determined by each evaluation metric. Experiments on three publicly available datasets show that, of 24 evaluation metrics, the traditional metric: area under receiver operating characteristic curve is the best metric for a CBIR application.     

A perceptual evaluation of grating frequencies and velocities in the analysis of dynamic images

Dynamic images that possess beauty and are user-friendly can increase the use of digital technology. In addition to information conveyance, dynamic images also act as a communication bridge in the human–machine interface. Dynamic images are widely used in the application of digital media. Therefore, understanding the visual effects of dynamic images on viewers is a very important issue. From a visual communication design perspective, dynamic images influence not only image quality, but also the viewers’ perception and impression of the displayed image. In the contemporary age characterized by universal usage of dynamic images, designers should attain synchronized knowledge and understanding of relevant media technology so as to present preferred design quality in the management of digital design such as animation design, Web page design, multimedia design, and so on. The current study noted that psychological effects such as viewers’ visual attention, preferences, and understanding were more important than image quality. Therefore, this study adopted the viewpoint of “visual design” and conducted a perceptual evaluation of grating frequency and grating velocity. The pair-comparison method and scale method were adopted in the research methodology to simplify perceptual evaluations and enhance their validity. The purpose of this study was: (1) To propose recommendations for displaying dynamic images and improving image performance using perceptual evaluation methods. (2) To examine the influence of psychological factors on viewer’s comfort when they experience dynamic images. (3) To identify the best grating feature combinations that meet viewer’s psychological characteristics and propose recommendations for dynamic images design. The study concludes that it is useful to establish criteria for evaluating users’ perception and to, develop perceptual evaluations of dynamic images. It is recommended that designers find a balance between watching a moving imaging display “clearly” and watching it “comfortably” for successful reception by the viewer.     

视觉注意力检测综述

人类能够迅速地选取视野中的关键部分,选择性地将视觉处理资源分配给这些视觉显著的区域.在计算机视觉领域,理解和模拟人类视觉系统的这种注意力机制,得到了学界的大力关注,并显示出了广阔的应用前景.近年来,随着计算能力的增强以及大规模显著性检测数据集的建立,深度学习技术逐渐成为视觉注意力机制计算和建模的主要手段.综述了视觉注意力检测的最新研究进展,包括人眼关注点检测和显著物体检测,并讨论了当前流行的视觉显著性检测数据集和常用的评估指标.对基于深度学习的工作进行了综述,也对之前代表性的非深度学习模型进行了讨论,同时,对这些模型在不同的数据集上的性能进行了详细评估.最后探讨了该领域的研究趋势和未来的发展方向.     

No-reference synthetic image quality assessment with convolutional neural network and local image saliency

Depth-image-based rendering (DIBR) is widely used in 3DTV, free-viewpoint video, and interactive 3D graphics applications. Typically, synthetic images generated by DIBR-based systems incorporate various distortions, particularly geometric distortions induced by object dis-occlusion. Ensuring the quality of synthetic images is critical to maintaining adequate system service. However, traditional 2D image quality metrics are ineffective for evaluating synthetic images as they are not sensitive to geometric distortion. In this paper, we propose a novel no-reference image quality assessment method for synthetic images based on convolutional neural networks, introducing local image saliency as prediction weights. Due to the lack of existing training data, we construct a new DIBR synthetic image dataset as part of our contribution. Experiments were conducted on both the public benchmark IRCCyN/IVC DIBR image dataset and our own dataset. Results demonstrate that our proposed metric outperforms traditional 2D image quality metrics and state-of-the-art DIBR-related metrics.

     

基于流形特征相似度的感知图像质量评价

图像质量评价（Image quality assessment, IQA）的目标是利用设计的计算模型得到与主观评价一致的结果,而人类视觉感知特性是感知图像质量评价的关键.大量研究发现,认知流形和拓扑连续性是人类感知的基础即人类感知局限在低维流形之上.基于图像低维流形特征分析,本文提出了基于流形特征相似度（Manifold feature similarity, MFS）的全参考图像质量评价方法.首先,利用正交局部保持投影算法来模拟大脑的视觉处理过程获取最佳映射矩阵进而得到图像的低维流形特征,通过流形特征的相似度来表征两幅图像的结构差异,从而反映感知质量上的差异.其次,考虑亮度失真对人眼视觉感知的影响,通过图像块均值计算亮度相似度并用于评价图像的亮度失真;最后,结合两个相似度得到图像的客观质量评价值.在四个公开图像测试库上的实验结果表明,所提出方法与现有代表性的图像质量方法相比总体上具有更好的评价结果.     

IAIR-CarPed: A psychophysically annotated dataset with fine-grained and layered semantic labels for object recognition

Unlike many other object recognition datasets which provide either category-level or within-category annotations, we introduce a novel dataset called “IAIR-CarPed” with layered semantic labels ranging from categories to fine-grained subcategories. These labels are collected from 20 subjects via strict psychophysical experiments. To the best of our knowledge, it is the first time that an object recognition dataset is built in this way to represent the adaptive and in-depth interpretations of objects in human vision. This dataset focuses on “car” and “pedestrian” which are two representative categories important in real applications. It contains 3132 images collected from pictures taken under various conditions and 8567 objects carefully annotated by all the 20 subjects. Besides fine-grained and layered semantic labels, five types of detailed visual difficulties of these objects are also provided, which can be adopted to evaluate the representation and generalization abilities of the recognition systems against individual difficulties. We present here the details of building this dataset, its statistics and properties, and then discuss possible applications of it with some primary experimental results.     

结合立体视觉舒适度的立体图像显著性检测

针对先前的立体图像显著性检测模型未充分考虑立体视觉舒适度和视差图分布特征对显著区域检测的影响,提出了一种结合立体视觉舒适度因子的显著性计算模型.该模型在彩色图像显著性提取中,首先利用SLIC算法对输入图像进行超像素分割,随后进行颜色相似区域合并后再进行二维图像显著性计算;在深度显著性计算中,首先对视差图进行预处理;然后基于区域对比度进行显著性计算;最后,结合立体视觉舒适度因子对二维显著图和深度显著图进行融合,得到立体图像显著图.在不同类型立体图像上的实验结果表明,该模型获得了85%的准确率和78%的召回率,优于现有常用的显著性检测模型,并与人眼立体视觉注意力机制保持良好的一致性.     

HVS模型及其在静止图象压缩质量评价中的应用

图象质量尺度是最优化图象压缩算法参数和提高图象质量的重要依据 .建立在人类视觉模型 (HVS)基础之上的感知质量尺度作为主客观联系的桥梁 ,能有效地反映出人对图象失真在视觉上的感知 .近年来 ,有许多研究者借助人类视觉系统研究中的最新成果 ,深入分析了与图象质量密切相关的视觉感知特性 ,提出了大量效果不错的静止图象压缩感知质量尺度 ,并对视觉感知特性在图象质量尺度中的应用方法进行了较全面的综述 ,揭示了影响其图象质量预测准确性、鲁棒性的主要因素 ,给出了该领域的最新研究成果和未来发展方向 .     

Quality assessment of 3D synthesized images based on structural and textural distortion

Depth image based rendering (DIBR) is a popular technique for rendering virtual 3D views in stereoscopic and autostereoscopic displays. The quality of DIBR-synthesized images may decrease due to various factors, e.g., imprecise depth maps, poor rendering techniques, inaccurate camera parameters. The quality of synthesized images is important as it directly affects the overall user experience. Therefore, the need arises for designing algorithms to estimate the quality of the DIBR-synthesized images. The existing 2D image quality assessment metrics are found to be insufficient for 3D view quality estimation because the 3D views not only contain color information but also make use of disparity to achieve the real depth sensation. In this paper, we present a new algorithm for evaluating the quality of DIBR generated images in the absence of the original references. The human visual system is sensitive to structural information; any deg radation in structure or edges affects the visual quality of the image and is easily noticeable for humans. In the proposed metric, we estimate the quality of the synthesized view by capturing the structural and textural distortion in the warped view. The structural and textural information from the input and the synthesized images is estimated and used to calculate the image quality. The performance of the proposed quality metric is evaluated on the IRCCyN IVC DIBR images dataset. Experimental evaluations show that the proposed metric outperforms the existing 2D and 3D image quality metrics by achieving a high correlation with the subjective ratings.

     

基于眼动数据的分类视觉注意模型

视觉注意是人类视觉系统中的重要部分,现有的视觉注意模型大多强调基于自底向上的注意,较少考虑自顶向下的语义,也鲜有针对不同类别图像的特定注意模型。眼动追踪技术可以客观、准确地捕捉到被试的注意焦点,但在视觉注意模型中的应用还比较少见。因此,提出了一种自底向上和自顶向下注意相结合的分类视觉注意模型CMVA,该模型针对不同类别的图像,在眼动数据的基础上训练分类视觉注意模型来进行视觉显著性预测。实验结果表明:与现有的其它8个视觉注意模型相比,该模型的性能最优。     

Visual comfort prediction for stereoscopic image using stereoscopic visual saliency

Perceptually salient regions of stereoscopic images significantly affect visual comfort (VC). In this paper, we propose a new objective approach for predicting VC of stereoscopic images according to visual saliency. The proposed approach includes two stages. The first stage involves the extraction of foreground saliency and depth contrast from a disparity map to generate a depth saliency map, which in turn is combined with 2D saliency to obtain a stereoscopic visual saliency map. The second stage involves the extraction of saliency-weighted VC features, and feeding them into a prediction metric to produce VC scores of the stereoscopic images. We demonstrate the effectiveness of the proposed approach compared with the conventional prediction methods on the IVY Lab database, with performance gain ranging from 0.016 to 0.198 in terms of correlation coefficients.     

Evaluation of visual saliency analysis algorithms in noisy images

Most existing visual saliency analysis algorithms assume that the input image is clean and does not have any disturbances. However, this situation is not always the case. In this paper, we provide an extensive evaluation of visual saliency analysis algorithms in noisy images. We analyze the noise immunity of saliency analysis algorithms by evaluating the performances of the algorithms in noisy images with increasing noise scales and by studying the effects of applying different denoising methods before performing saliency analysis. We use 10 state-of-the-art saliency analysis algorithms and 7 typical image denoising methods on 4 eye fixation datasets and 2 salient object detection datasets. Our experiments show that the performances of saliency analysis algorithms decrease with increasing image noise scales in general. An exception is that the nonlinear features (NF) integrated algorithm shows good noise immunity. We also find that image denoising methods can greatly improve the noise immunity of the algorithms. Our results show that the combination of NF and Median denoising method works best on eye fixation datasets and the combination of saliency optimization (SO) and color block-matching and 3D filtering (C-BM3D) method works best on salient object detection datasets. The combination of SO and Average denoising method works best for applications wherein time efficiency is a major concern for both types of datasets.     

Leveraging colour-based pseudo-labels to supervise saliency detection in hyperspectral image datasets

Saliency detection mimics the natural visual attention mechanism that identifies an imagery region to be salient when it attracts visual attention more than the background. This image analysis task covers many important applications in several fields such as military science, ocean research, resources exploration, disaster and land-use monitoring tasks. Despite hundreds of models have been proposed for saliency detection in colour images, there is still a large room for improving saliency detection performances in hyperspectral imaging analysis. In the present study, an ensemble learning methodology for saliency detection in hyperspectral imagery datasets is presented. It enhances saliency assignments yielded through a robust colour-based technique with new saliency information extracted by taking advantage of the abundance of spectral information on multiple hyperspectral images. The experiments performed with the proposed methodology provide encouraging results, also compared to several competitors.

     

超像素条件随机场下的RGB-D视频显著性检测

目的 视觉显著性在众多视觉驱动的应用中具有重要作用,这些应用领域出现了从2维视觉到3维视觉的转换,从而基于RGB-D数据的显著性模型引起了广泛关注。与2维图像的显著性不同,RGB-D显著性包含了许多不同模态的线索。多模态线索之间存在互补和竞争关系,如何有效地利用和融合这些线索仍是一个挑战。传统的融合模型很难充分利用多模态线索之间的优势,因此研究了RGB-D显著性形成过程中多模态线索融合的问题。方法 提出了一种基于超像素下条件随机场的RGB-D显著性检测模型。提取不同模态的显著性线索,包括平面线索、深度线索和运动线索等。以超像素为单位建立条件随机场模型,联合多模态线索的影响和图像邻域显著值平滑约束,设计了一个全局能量函数作为模型的优化目标,刻画了多模态线索之间的相互作用机制。其中,多模态线索在能量函数中的权重因子由卷积神经网络学习得到。结果 实验在两个公开的RGB-D视频显著性数据集上与6种显著性检测方法进行了比较,所提模型在所有相关数据集和评价指标上都优于当前最先进的模型。相比于第2高的指标,所提模型的AUC（area under curve）,sAUC（shuffled AUC）,SIM（similarity）,PCC（Pearson correlation coefficient）和NSS（normalized scanpath saliency）指标在IRCCyN数据集上分别提升了2.3%,2.3%,18.9%,21.6%和56.2%;在DML-iTrack-3D数据集上分别提升了2.0%,1.4%,29.1%,10.6%,23.3%。此外还进行了模型内部的比较,验证了所提融合方法优于其他传统融合方法。结论 本文提出的RGB-D显著性检测模型中的条件随机场和卷积神经网络充分利用了不同模态线索的优势,将它们有效融合,提升了显著性检测模型的性能,能在视觉驱动的应用领域发挥一定作用。     

视觉感知正反馈的显著性检测

目的 人类视觉系统性能远超当前机器视觉,模拟人类视觉机制改进当前算法是有效研究途径,为此提出一种视觉感知正反馈模型,通过循环迭代、重复叠加视觉刺激生成更符合人类感知的视觉显著性图。方法 首先用多种常规方法检测图像显著度,模拟人类视觉多通道特性,再组合这些显著图为综合显著图;利用显著度大的像素构建初始注视区。其次借助集成RVFL（随机向量功能网络）模拟人脑神经网络产生视觉刺激,对注视与非注视区内像素在线“随机采样—学习建模”,图像像素经模型分类获得新注视区。对新注视区与非注视区,可重复迭代进行“随机采样—学习建模—像素分类”;迭代中若注视区连续相同,则表明感知饱和,迭代终止。若将每次像素分类结果看做是一种视觉刺激,则多次视觉刺激输出叠加,可生成新的图像显著性图。最终的像素分类结果就是图像分割目标。结果 将本文算法与现有方法在标准图像数据库上进行对比评测,包括通过对6种算法在ECSSD、SED2和MSRA10K 3个图像数据库上的P-R曲线,F-measure值和平均绝对误差（MAE）值上进行定量分析,对6种模型生成的显著性图作定性比较。数据表明,本文算法在SED2和MSRA10K图象数据库中性能最好,在ECSSD图象数据库中稍低于BL（bootstrap learning）和RBD（robust background detection）算法。本文算法的显著图与人类视觉感知更接近。且算法的正反馈迭代过程一般可迅速饱和,并未显著增加算法负担。实验结果表明,本文方法可作为一种有效的后处理手段,显著提升常规显著性检测算法的性能。结论 提出了一种模拟人类视觉机制的数据驱动显著性检测算法,无需图像先验知识和事先的标记样本。面对多目标,背景复杂等情况,本文方法具有相对好的鲁棒性和适用性,并且能够较好解决现实环境中图像处理算法的通用性、可靠性和准确性问题。