An Objective Video Quality Metric for Compressed Stereoscopic Video

Stereoscopic video create the impression of depth by using two slightly different viewpoints. Conventional video quality assessment methods for 2D video are not suitable for stereoscopic video, so a new video quality assessment model for stereoscopic video is needed. In this paper, we propose a new objective video quality metric for compressed stereoscopic video. The proposed algorithm uses blocking artifacts, blurring in edge regions and video quality difference between two views. Blocking artifacts and blurring in edge regions are distortion appeared in the compressed video, and they are widely used in conventional video quality model. Difference in video quality between two views considers 3D effect of stereoscopic video. To verify the performance of the proposed algorithm, we performed subjective evaluation of stereoscopic video, and we confirmed that the proposed algorithm is superior to the conventional algorithms in respect to correlation with the subjective evaluation.     

Post‐processing of 3D Video Extension of H.264/AVC for a Quality Enhancement of Synthesized View Sequences

Since July of 2012, the 3D video extension of H.264/AVC has been under development to support the multi‐view video plus depth format. In 3D video applications such as multi‐view and free‐view point applications, synthesized views are generated using coded texture video and coded depth video. Such synthesized views can be distorted by quantization noise and inaccuracy of 3D wrapping positions, thus it is important to improve their quality where possible. To achieve this, the relationship among the depth video, texture video, and synthesized view is investigated herein. Based on this investigation, an edge noise suppression filtering process to preserve the edges of the depth video and a method based on a total variation approach to maximum a posteriori probability estimates for reducing the quantization noise of the coded texture video. The experiment results show that the proposed methods improve the peak signal‐to‐noise ratio and visual quality of a synthesized view compared to a synthesized view without post processing methods.     

A new sharpness metric based on local kurtosis,edge and energy information

Sharpness metrics that use the whole frequency spectrum of the image cannot separate the sharpness information from the scene content. The sharpness metrics that use spatial gradients of the edges work only for comparisons among versions of the same image. We have developed a content independent, no-reference sharpness metric based on the local frequency spectrum around the image edges. In this approach, we create an edge profile by detecting edge pixels and assigning them to 8×8 pixel blocks. Then we compute sharpness using the average 2D kurtosis of the 8×8 DCT blocks. However, average kurtosis is highly sensitive to asymmetry in the DCT, e.g. different amounts of energy and edges in the x and y directions, therefore causing problems with different content and asymmetric sharpness enhancement. Thus we compensate the kurtosis using spatial edge extent information and the amount of vertical and horizontal energy in the DCT. The results show high correlation with subjective quality for sharpness-enhanced video and high potential to deal with asymmetric enhancement. For compressed, extremely sharpened and noisy video, the metric correlates with subjective scores up to the point where impairments become strongly noticeable in the subjective quality evaluation. The metric can be used by itself as a control variable for high-quality image capture and display systems, high-quality sharpness enhancement algorithms, and as a key component of a more general overall quality metric.     

A no-reference sharpness metric based on the notion of relative blur for Gaussian blurred image

This work presents a no-reference sharpness metric for Gaussian blurred image. The metric is based on the notion of relative blur. The key concept is that the judgement on the sharpness closely relates to the degree of convenience for recognizing image objects on a certain scale. Based on this concept, the proposed metric is defined as relative blur with respect to certain object scale using an absolute blur measure. The object scale is characterized by a granularity analysis of image content. And the absolute blur is built on an analysis of edge local gray level distribution. The performance of the metric is tested and compared with some outstanding existing metrics in this field on three widely used databases. The experiment results show that the proposed metric can predict the sharpness of images in varying databases with high accuracy and reliability.     

基于稀疏表示的立体图像客观质量评价方法

提出了一种基于稀疏表示的立体图像质量评价方法 ,分为训练和测试两个部分。在训练部 分,通过训练不同频带的立体图像获得立体图像的稀疏字典;在测试部分,根据稀疏字典计 算得到立体图 像的稀疏特征,定义了稀疏特征相似度衡量原始和失真图像信息的差异,并根据稀疏字典计 算了频带增益和左右视点的融合权值,最后融合稀疏特征相似度作为立体图像质量的 客观评价值。在立体图像测试库上的实验结果表明,本文方法的评价结果与主观评价结果有 较好的相关性,符合人类视觉系统的感知。     

Depth map misalignment correction and dilation for DIBR view synthesis

The quality of the synthesized views by Depth Image Based Rendering (DIBR) highly depends on the accuracy of the depth map, especially the alignment of object boundaries of texture image. In practice, the misalignment of sharp depth map edges is the major cause of the annoying artifacts at the disoccluded regions of the synthesized views. Conventional smooth filter approach blurs the depth map to reduce the disoccluded regions. The drawbacks are the degradation of 3D perception of the reconstructed 3D videos and the destruction of the texture in background regions. Conventional edge preserving filter utilizes the color image information in order to align the depth edges with color edges. Unfortunately, the characteristics of color edges and depth edges are very different which causes annoying boundaries artifacts in the synthesized virtual views. Recent solution of reliability-based approach uses reliable warping information from other views to fill the holes. However, it is not suitable for the view synthesis in video-plus-depth based DIBR applications. In this paper, a new depth map preprocessing approach is proposed. It utilizes Watershed color segmentation method to correct the depth map misalignment and then the depth map object boundaries are extended to cover the transitional edge regions of color image. This approach can handle the sharp depth map edges lying inside or outside the object boundaries in 2D sense. The quality of the disoccluded regions of the synthesized views can be significantly improved and unknown depth values can also be estimated. Experimental results show that the proposed method achieves superior performance for view synthesis by DIBR especially for generating large baseline virtual views.     

Adaptive depth truncation filter for MVC based compressed depth image

In multiview video plus depth (MVD) format, virtual views are generated from decoded texture videos with corresponding decoded depth images through depth image based rendering (DIBR). 3DV-ATM is a reference model for the H.264/AVC based multiview video coding (MVC) and aims at achieving high coding efficiency for 3D video in MVD format. Depth images are first downsampled then coded by 3DV-ATM. However, sharp object boundary characteristic of depth images does not well match with the transform coding based nature of H.264/AVC in 3DV-ATM. Depth boundaries are often blurred with ringing artifacts in the decoded depth images that result in noticeable artifacts in synthesized virtual views. This paper presents a low complexity adaptive depth truncation filter to recover the sharp object boundaries of the depth images using adaptive block repositioning and expansion for increasing the depth values refinement accuracy. This new approach is very efficient and can avoid false depth boundary refinement when block boundaries lie around the depth edge regions and ensure sufficient information within the processing block for depth layers classification. Experimental results demonstrate that the sharp depth edges can be recovered using the proposed filter and boundary artifacts in the synthesized views can be removed. The proposed method can provide improvement up to 3.25 dB in the depth map enhancement and bitrate reduction of 3.06% in the synthesized views.     

Reduced Reference Quality Metric for Synthesized Virtual Views in 3DTV

Multi‐view video plus depth (MVD) has been widely used owing to its effectiveness in three‐dimensional data representation. Using MVD, color videos with only a limited number of real viewpoints are compressed and transmitted along with captured or estimated depth videos. Because the synthesized views are generated from decoded real views, their original reference views do not exist at either the transmitter or receiver. Therefore, it is challenging to define an efficient metric to evaluate the quality of synthesized images. We propose a novel metric—the reduced‐reference quality metric. First, the effects of depth distortion on the quality of synthesized images are analyzed. We then employ the high correlation between the local depth distortions and local color characteristics of the decoded depth and color images, respectively, to achieve an efficient depth quality metric for each real view. Finally, the objective quality metric of the synthesized views is obtained by combining all the depth quality metrics obtained from the decoded real views. The experimental results show that the proposed quality metric correlates very well with full reference image and video quality metrics.     

Toward an unsupervised blind stereoscopic 3D image quality assessment using joint spatial and frequency representations

Existing blind stereoscopic 3D (S3D) image quality assessment (IQA) metrics usually require supervised learning methods to predict S3D image quality, which limits their applicability in practice. In this paper, we propose an unsupervised blind S3D IQA metric that utilizes the joint spatial and frequency representations of visual perception. The metric proposed in this work was inspired by the binocular visual mechanism; furthermore, it is unsupervised and does not require subject-rated samples for training. To be more specific, first, the various binocular quality-aware features in spatial and frequency domains are extracted from the monocular and cyclopean views of natural S3D image patches. Subsequently, these features are utilized to establish a pristine multivariate Gaussian (MVG) model to characterize natural S3D image regularities. Finally, with the learned MVG model, the final quality score for a distorted S3D image can be yielded using a Bhattacharyya-like distance. Our experimental results illustrate that, compared to related existing metrics, the devised metric achieves competitive prediction performance.     

图像质量快速盲检测及其在视觉系统中的应用

提出了一种新的图像质量快速盲检测方法,以锐利度作为图像质量准则,而锐利度可以归结为图像的模糊特性,并由边缘信息估计得到。为了提高系统效率,采用边缘对比度的估计方法,并以边缘对比度作为边缘锐利度的判据准则。分析了邻近边缘的相互影响,因而只需从有限的边缘点来提取出合理的线扩展函数(LSF),进而得到模糊参数。在LSF计算中,采用一种新的插值计算方法。实验结果表明:当图像不是特别模糊时,测量结果是准确的。     

基于三维感知的立体虚拟视点图像质量评价方法

为了评价立体虚拟视点图像的质量,提出了一种基 于三维感知的客观评价方法。综合考虑了立体虚拟视点图像两大最主要失真类型:单视点绘 制失真和立体视点不匹配失真。针对单视点绘制失真,先提取 当前视点失真图与无失真图的差异性区域,再针对该差异性区域计算平均结构相似度(MSSIM ),最后将左 右视点平均池化作为单目纹理特征值;针对立体视点不匹配失真,先对左右视点失真图分别 进行视差映射, 再提取映射图与该视点失真图的差异区域作为双目不匹配区域,然后针对不匹配区域计算MS SIM 值,最 后将左右视点平均池化作为双目竞争特征值;最终将两个特征值幂次融合,作为最终的立体 虚拟视点图像 质量评价客观指标。实验结果表明本方法有效匹配主观打分的DMOS值,皮尔森线性相关系数 和斯皮尔曼 秩相关系数分别为0.911和0.900,正确反映了 立体虚拟视点图像质量。     

Quality assessment of multiply and singly distorted stereoscopic images via adaptive construction of cyclopean views

A challenging problem confronted when designing a blind/no-reference (NR) stereoscopic image quality assessment (SIQA) algorithm is to simulate the quality assessment (QA) behavior of the human visual system (HVS) during binocular vision. An effective way to solve this problem is to estimate the quality of the merged single view created in the human brain which is also referred to as the cyclopean image. However, due to the difficulty in modeling the binocular fusion and rivalry properties of the HVS, obtaining effective cyclopean images for QA is non-trivial, and consequently previous NR SIQA algorithms either require the MOS/DMOS values of the distorted 3D images for training or ignore the quality analysis of the merged cyclopean view. In this paper, we focus on (1) constructing accurate and appropriate cyclopean views for QA of stereoscopic images by adaptively analyzing the distortion information of two monocular views, and (2) training NR SIQA models without requiring the assistance of the MOS/DMOS values in existing databases. Accordingly, we present an effective opinion-unaware SIQA algorithm called MUSIQUE-3D, which blindly assesses the quality of multiply and singly distorted stereoscopic images by analyzing quality degradations of both monocular and cyclopean views. The monocular view quality is estimated by an extended version of the MUSIQUE algorithm, and the cyclopean view quality is computed from the distortion parameter values predicted by a two-layer classification-regression model trained on a large 3D image dataset. Tests on various 3D image databases demonstrate the superiority of our method as compared with other state-of-the-art SIQA algorithms.     

No-reference Stereoscopic Image Quality Assessment Using Binocular Self-similarity and Deep Neural Network

Quality assessment of three-dimensional (3D) images is more challenging than that of 2D images. The quality of 3D visual experience is one of the most challenging areas of human binocular perception and is affected by multiple factors such as asymmetric stereo image/video compression, depth perception, visual discomfort, and single view quality. In this paper, we propose a new no-reference quality assessment method for stereoscopic images based on Binocular Self-similarity (BS) and Deep Neural Networks (DNN). To be more specific, a BS index is defined and computed according to binocular rivalry and suppression based on the depth image-based rendering technique. Then, a DNN is trained in an opinion unaware way to predict local quality. Binocular integration (BI) index is calculated by using the trained DNN, accounting for binocular integration behaviors. Finally, the final quality score of stereoscopic image is obtained by combining the BS and BI indexes together. Experimental results on four public 3D image quality assessment databases demonstrate that compared with existing methods, the proposed method can achieve high consistency with subjective perception on stereoscopic images with both symmetric and asymmetric distortions.     

A perceptual stereoscopic image quality assessment model accounting for binocular combination behavior

Stereoscopic image quality assessment (SIQA) plays an important role in the development of 3D image processing. In this paper, a full-reference object SIQA model is built based on binocular summation channel and binocular difference channel. In our frame work, binocular combination behavior and how to experience the depth perception are thought to be the key factors to evaluate the quality of stereoscopic images. Differing from the current depth map methods, this method focuses on a new aspect, and an effective combination model is proposed based on the physiological findings in the Human Visual System (HVS). Experimental results demonstrate that the proposed quality assessment metric significantly outperforms the existing metrics and can achieve higher consistency with subject quality assessment when predicting the quality of stereoscopic images that have been symmetrically distorted.     

基于双目特征联合的无参考立体图像质量评价

通过模拟人类视觉系统(HVS)的双目视觉行为,提 出一种基于双目特征联合的无参考立 体图像质量评价(NR-SIQA)方法。首先分析立体视觉感知中的双目联合行为,提出 可应用于立体图像质量预 测的双目联合模型;然后采用学习和统计分析的方法,分别提取局部和全局特征并联合作 为感知特征; 最后采用机器学习算法,建立特征和质量的关系模型,并结合基于特征的双目联合模型预测 立体图像质量。实验结果表明,本文方法在对称立体图像库上的Pearson线性相关系数(PLCC)和Spearman等级系数(SRCC)高于0.93,在非对称库上高于0.87,优 于现有评价方法。     

Adaptive enhancement of compressed SAR images

Synthetic aperture radar (SAR) imagery is an important global all-weather surveillance and mapping satellite imagery system. As space-borne systems have a limited storage capacity, it is imperative to heavily compress SAR images, possible with lossy compression schemes. As a result, SAR images need to be enhanced in earth stations. The work reported in this paper aims to address the issue of compression artefact removal of SAR images in an adaptive manner. The SAR images, compressed using the JPEG utility at significantly low bit rates, are enhanced by adaptively removing coding artefacts and speckle noise. As edges carry significant information in satellite imagery, a significant edge image is used for edge enhancement with selective removal of noisy edges. Further, an image sharpness metric is proposed in this work to serve as an objective no-reference metric for measuring the sharpness of SAR images.     

View upsampling optimization for mixed resolution 3D video coding

3D video is composed out of two or more, temporally synchronized, 2D video streams acquired at different camera poses and accompanied by geometrical information. In a mixed resolution 3D video stream, a subset of views is coded at reduced resolution. It has been shown in the literature that subjective quality of mixed resolution 3D video is close to that of full resolution 3D video. In order to improve the coding gain in mixed resolution coding scenario we present a new depth encoding method called view upsampling optimization. A novel depth distortion metric based on the performance of the depth-based super resolution is also presented. Finally, to improve the quality of the decoded video an improved depth-based super resolution method that uses view synthesis quality mapping is used for upsampling of low resolution views. The simulations, performed with the recently standardized MVC+D encoder, show that the proposed solution combined with the state of the art view synthesis distortion outperforms the anchor MVC+D coding scheme by 14.5 % of dBR on average for the total coded bitrate and by 17 % of dBR on average for the synthesized views.     

End-to-end stereoscopic video streaming with content-adaptive rate and format control

We address efficient compression and real-time streaming of stereoscopic video over the current Internet. We first propose content-adaptive stereo video coding (CA-SC), where additional coding gain, over that can be achieved by exploiting only inter-view correlations, is targeted by down-sampling one of the views spatially or temporally depending on the content, based on the well-known theory that the human visual system can perceive high frequencies in three-dimensional (3D) from the higher quality view. We also developed stereoscopic 3D video streaming server and clients by modifying available open source platforms, where each client can view the video in mono or stereo mode depending on its display capabilities. The performance of the end-to-end stereoscopic streaming system is demonstrated using subjective quality tests.     

交互式立体显示中基于图像拼合的视点合成

在立体显示中,视点合成是实现交互性的关键技术,即在三维(3D)场景中通过自由选择视点而获得环视能力.本文将视点插值和基于图像拼合的视点变形技术相结合,提出一种中间视合成算法.首先均匀化原始立体图像对;然后只对前景对象区域进行视差估计以提高视差匹配的速度和精度;接着确定左右视点中的可靠区域,根据可靠区域生成过渡中间视点;最后,采用视点插值结合变形的方法,由过渡视合成中间视点.实验结果表明合成的中间视点图像质量良好,而且合成速度也明显提高.本文算法可用于实时 3D 视频应用的交互式立体显示,可以实现任意视点的实时绘制.     

Mask topography effects in projection printing of phase-shiftingmasks

Topography effects of glass edges in phase-shifting masks (PSM's) on image quality are assessed using the rigorous electromagnetic simulation program TEMPEST on three different optical systems for four PSM technologies including alternating, rim, attenuated, and chromeless. The scalar and thin mask approximations used in simulation programs such as SPLAT can be in error by as much as 20% for certain classes of shifter edges. A feature size independent bias of 0.021 λ/NA per edge is recommended for alternating masks with vertical edges because light is lost near the etched glass edges. No direct electromagnetic interaction between chromium edges and shifter edges was found for rim phase-shifting masks. The rim dimension can thus be designed solely on the basis of the sidelobe level and peak intensity. For attenuated PSM, edge effects are less severe but sidelobe problems occur. For a center to sidelobe contrast of 0.6 over a DOF of 3 RU, a lower transmission of 4% is recommended. For chromeless PSM, the imbalance in image peaks is shown to be affected by the optical stepper parameters. In any PSM technology, it appears that a 360° glass protrusion may produce a drastic drop in intensity due to resonant effects