Multiview depth coding based on combined color/depth segmentation

In this paper, a new coding method for multiview depth video is presented. Considering the smooth structure and sharp edges of depth maps, a segmentation based approach is proposed. This allows further preserving the depth contours thus introducing fewer artifacts in the depth perception of the video. To reduce the cost associated with partition coding, an approximation of the depth partition is built using the decoded color view segmentation. This approximation is refined by sending some complementary information about the relevant differences between color and depth partitions. For coding the depth content of each region, a decomposition into orthogonal basis is used in this paper although similar decompositions may be also employed. Experimental results show that the proposed segmentation based depth coding method outperforms H.264/AVC and H.264/MVC by more than 2 dB at similar bitrates.     

Overview of the MVC + D 3D video coding standard

3D video services are emerging in various application domains including cinema, TV broadcasting, Blu-ray discs, streaming and smartphones. A majority of the 3D video content in market is still based on stereo video, which is typically coded with the multiview video coding (MVC) extension of the Advanced Video Coding (H.264/AVC) standard or as frame-compatible stereoscopic video. However, the 3D video technologies face challenges as well as opportunities to support more demanding application scenarios, such as immersive 3D telepresence with numerous views and 3D perception adaptation for heterogeneous 3D devices and/or user preferences. The Multiview Video plus Depth (MVD) format enables depth-image-based rendering (DIBR) of additional viewpoints in the decoding side and hence helps in such advanced application scenarios. This paper reviews the MVC + D standard, which specifies an MVC-compatible MVD coding format.     

Multiview video compression with 1-D transforms

Many alternative transforms have been developed recently for improved compression of images, intra prediction residuals or motion-compensated prediction residuals. In this paper, we propose alternative transforms for multiview video coding. We analyze the spatial characteristics of disparity-compensated prediction residuals, and the analysis results show that many regions have 1-D signal characteristics, similar to previous findings for motion-compensated prediction residuals. Signals with such characteristics can be transformed more efficiently with transforms adapted to these characteristics and we propose to use 1-D transforms in the compression of disparity-compensated prediction residuals in multiview video coding. To show the compression gains achievable from using these transforms, we modify the reference software (JMVC) of the multiview video coding amendment to H.264/AVC so that each residual block can be transformed either with a 1-D transform or with the conventional 2-D Discrete Cosine Transform. Experimental results show that coding gains ranging from about 1–15% of Bjontegaard-Delta bitrate savings can be achieved.     

Depth map misalignment correction and dilation for DIBR view synthesis

The quality of the synthesized views by Depth Image Based Rendering (DIBR) highly depends on the accuracy of the depth map, especially the alignment of object boundaries of texture image. In practice, the misalignment of sharp depth map edges is the major cause of the annoying artifacts at the disoccluded regions of the synthesized views. Conventional smooth filter approach blurs the depth map to reduce the disoccluded regions. The drawbacks are the degradation of 3D perception of the reconstructed 3D videos and the destruction of the texture in background regions. Conventional edge preserving filter utilizes the color image information in order to align the depth edges with color edges. Unfortunately, the characteristics of color edges and depth edges are very different which causes annoying boundaries artifacts in the synthesized virtual views. Recent solution of reliability-based approach uses reliable warping information from other views to fill the holes. However, it is not suitable for the view synthesis in video-plus-depth based DIBR applications. In this paper, a new depth map preprocessing approach is proposed. It utilizes Watershed color segmentation method to correct the depth map misalignment and then the depth map object boundaries are extended to cover the transitional edge regions of color image. This approach can handle the sharp depth map edges lying inside or outside the object boundaries in 2D sense. The quality of the disoccluded regions of the synthesized views can be significantly improved and unknown depth values can also be estimated. Experimental results show that the proposed method achieves superior performance for view synthesis by DIBR especially for generating large baseline virtual views.     

View-spatial–temporal post-refinement for view synthesis in 3D video systems

Depth image based rendering is one of key techniques to realize view synthesis for three-dimensional television and free-viewpoint television, which provide high quality and immersive experiences to end viewers. However, artifacts of rendered images, including holes caused by occlusion/disclosure and boundary artifacts, may degrade the subjective and objective image quality. To handle these problems and improve the quality of rendered images, we present a novel view-spatial–temporal post-refinement method for view synthesis, in which new hole-filling and boundary artifact removal techniques are proposed. In addition, we propose an optimal reference frame selection algorithm for a better trade-off between the computational complexity and rendered image quality. Experimental results show that the proposed method can achieve a peak signal-to-noise ratio gain of 0.94 dB on average for multiview video test sequences when compared with the benchmark view synthesis reference software. In addition, the subjective quality of the rendered image is also improved.     

Recent MPEG Standardization Activities on 3D Video Coding

The Moving Picture Experts Group(MPEG) has been developing a 3D video(3DV) coding standard for depth-based 3DV data representations,especially for multiview video plus depth(MVD) format.With MVD,depth-image-based rendering(DIBR) is used to synthesize virtual views that are based on a few transmitted pairs of texture and depth data.In this paper,we discuss ongoing 3DV standardization and summarize coding tools proposed in the responses to MPEG’s call for proposals on 3DV coding.     

结合预处理的深度视频帧内快速编码算法

多视点彩色加深度(MVD)视频是三维(3D)视频的 主流格式。在3D高效视频编码中,深度视频帧内编码 具有较高的编码复杂度;深度估计软件获取的深度视频由于不够准确会使深度图平坦 区域纹理增加, 从而进一步增加帧内编码复杂度。针对以上问题,本文提出了一种联合深度处理的深度视频 帧内低复杂度 编码算法。首先,在编码前对深度视频进行预处理,减少由于深度图不准确而出现的纹理信 息;其次,运 用反向传播神经网络(BPNN,backpropagation neural network)预测最大编码单元 (LCU,la rgest coding unit)的最大划分深度;最后联合深度视频的边缘信 息及对应的彩色LCU最大划分深度进行CU提前终止划分和快速模式选取。实验结果表明, 本文算法在保证 虚拟视点质量的前提下,BDBR下降0.33% ,深度视频编码时间平均节省50.63%。     

Key-frame-based depth propagation for semi-automatic stereoscopic video conversion

In this paper, we propose a key-frame-based bi-directional depth propagation algorithm for semi-automatic 2D-to-3D stereoscopic video conversion. First, key-frames are identified from each video shot based on color motion-compensation errors to prevent high-motion content between any pair of consecutive key frames. Depths for key-frames are manually assigned or rendered by popular computer tools, and then bi-directionally propagated to non-key-frames there between. Our depth propagation algorithm is featured of a multi-pass error correcting procedure for each frame to prevent depth artifacts from being further propagated to adjacent frames. Our proposed algorithm is advantageous in solving the background occlusion/dis-occlusion problem that degrades the performances of traditional depth propagation algorithms. Experimental results show that our scheme is capable of achieving better results against three prior algorithms in view of the qualities of the estimated depth map (e.g., dis-occluded background and object boundaries) and the synthesized stereo views.     

一种用于多视视频加深度联合编码的错误隐藏算法

ATM(AVC-based test model)测试模型实现了多视视频加深度(MVD)格式的联合编码,使得数据的压缩效率更高。然而,较高的压缩效率使得码流对传输错误非常敏感,极易产生错误扩散现象。针对ATM测试模型的编码顺序,提出一种用于MVD联合编码的错误隐藏算法。算法充分利用视点内、视点间及纹理视频与深度视频间的相关性,针对每个视点的不同特征提出了适应其视频特性的不同隐藏算法。实验表明,本文提出的算法可以在不增加算法复杂度的情况下,有效提高视频的主客观质量。     

Reduced Reference Quality Metric for Synthesized Virtual Views in 3DTV

Multi‐view video plus depth (MVD) has been widely used owing to its effectiveness in three‐dimensional data representation. Using MVD, color videos with only a limited number of real viewpoints are compressed and transmitted along with captured or estimated depth videos. Because the synthesized views are generated from decoded real views, their original reference views do not exist at either the transmitter or receiver. Therefore, it is challenging to define an efficient metric to evaluate the quality of synthesized images. We propose a novel metric—the reduced‐reference quality metric. First, the effects of depth distortion on the quality of synthesized images are analyzed. We then employ the high correlation between the local depth distortions and local color characteristics of the decoded depth and color images, respectively, to achieve an efficient depth quality metric for each real view. Finally, the objective quality metric of the synthesized views is obtained by combining all the depth quality metrics obtained from the decoded real views. The experimental results show that the proposed quality metric correlates very well with full reference image and video quality metrics.     

多视点与深度视频编码技术研究综述

多视点与深度视频(MVD)可以有效表示自由视点视频,减少了需传输视点的个数,该表达形式正受到越来越多的关注,如何对MVD进行高效编码尤为重要.介绍了自由视点视频的发展现状及国内外研究概况,详细讨论了深度图像的高效编码技术及多视点与深度联合编码技术,并对多视点与深度视频编码技术进行了总结及展望.     

Iterative search strategy with selective bi-directional prediction for low complexity multiview video coding

The multiview video coding (MVC) extension of H.264/AVC is the emerging standard for compression of impressive 3D and free-viewpoint video. The coding structure in MVC adopts motion and disparity estimation to exploit temporal and inter-view dependencies in MVC. It results in a considerable increase in encoding complexity. Most of the computational burden comes from uni-directional and bi-directional prediction. In this paper, an iterative search strategy is designed to speed up the uni-directional prediction in MVC. It can work with an adaptive search range adjustment through a confidence measure of a loop constraint to obtain both motion and disparity vectors jointly. Furthermore, a selective bi-directional prediction algorithm is proposed to enhance the coding performance by analyzing the statistical characteristics of bi-directional prediction in MVC. Experimental results demonstrate that, by using the proposed fast search, the temporal and inter-view redundancies of multiview video can be eliminated sufficiently with low complexity.     

用于多视视频加深度的错误隐藏算法

提出了一种针对多视视频加深度(MVD)的错误隐藏( EC)算法。算法充分利用MVD特有的深度 信息及当前丢失宏块周围正确解码的宏块信息,将丢失宏块分为3类不同属性的宏块。针对3类宏块的特 点,分别提出了基于候选运动矢量修正(CMVR)、基于深度的外边界匹配(DOBMA)以及自适应 权 值的EC(AWEC)等模式。实验表明,本文提出的算法在保证相同的视频主客观质量情况下, 能够快速有效地实现EC。     

Disparity field and depth map coding for multiview 3D image generation

In the present paper techniques are examined for the coding of the depth map and disparity fields for stereo or multiview image communication applications. It is assumed that both the left and right channels of the multiview image sequence are coded using block- or object-based methods. A dynamic programming algorithm is used to estimate a disparity field between each stereo image pair. Depth is then estimated and occlusions are optionally detected, based on the estimated disparity fields. Spatial interpolation techniques are examined based on the disparity/depth information and the detection of occluded regions using either stereoscopic or trinocular camera configurations. It is seen that the presence of a third camera at the transmitter site improves the estimation of disparities, the detection of occlusions and the accuracy of the resulting spatial interpolation at the receiver. Various disparity field and depth map coding techniques are then proposed and evaluated, with emphasis given to the quality of the resulting intermediate images at the receiver site. Block-based and wireframe modeling techniques are examined for the coding of isolated depth or disparity map information. Further, 2D and 3D motion compensation techniques are evaluated for the coding of sequences of depth or disparity maps. The motion fields needed may be available as a byproduct of block-based or object-based coding of the intensity images. Experimental results are given for the evaluation of the performance of the proposed coding and spatial interpolation methods.     

Depth reconstruction filter for depth coding

Depth images represent the distances of scene elements from a camera in 3D space; their efficient coding is crucial for emerging applications such as free-viewpoint TV and 3D video. An in-loop reconstruction filter that improves the depth-coding performance as well as the rendering quality of virtual views based upon the coded depth is proposed.     

基于Kinect的实时深度提取与多视绘制算法

提出了一种基于Kinect的实时深度提取算法和单纹理+深度的多视绘制方法。在采集端,使用Kinect提取场景纹理和深度,并针对Kinect输出深度图的空洞提出一种快速修复算法。在显示端,针对单纹理+深度的基于深度图像的绘制(DIBR,depth image based rendering)绘制产生的大空洞,采用一种基于背景估计和前景分割的绘制方法。实验结果表明,本文方法可实时提取质量良好的深度图,并有效修复了DIBR绘制过程中产生的大空洞,得到质量较好的多路虚拟视点图像。以所提出的深度获取和绘制算法为核心,实现了一种基于深度的立体视频系统,最终的虚拟视点交织立体显示的立体效果良好,进一步验证了本文算法的有效性。本文系统可用于实景的多视点立体视频录制与播放。     

3D-HEVC中深度图编码技术研究进展

3D-HEVC是为了满足3D视频和自由视点视频的高效编码而最新制定的视频编码标准,它要求同时编码几个视点的纹理视频和深度图.完全采用传统的技术来编码深度图会使得深度图内部锐利边界处产生伪影效应,为此,一些新的针对于深度图的编码工具被开发.详细介绍了这些编码工具,同时介绍了编码深度图时所使用的率失真优化方法.     

Adaptive Multiview Video Coding Scheme Based on Spatiotemporal Correlation Analyses

In this paper, we propose an adaptive multiview video coding scheme based on spatiotemporal correlation analyses using hierarchical B picture (AMVC‐HBP) for the integrative encoding performances, including high compression efficiency, low complexity, fast random access, and view scalability, by integrating multiple prediction structures. We also propose an in‐coding mode‐switching algorithm that enables AMVC‐HBP to adaptively select a better prediction structure in the encoding process without any additional complexity. Experimental results show that AMVC‐HBP outperforms the previous multiview video coding scheme based on H.264/MPEG‐4 AVC using the hierarchical B picture (MVC‐HBP) on low complexity for 21.5%, on fast random access for about 20%, and on view scalability for 11% to 15% on average. In addition, distinct coding gain can be achieved by AMVC‐HBP for dense and fast‐moving sequences compared with MVC‐HBP.     

Reconstruction of lost depth data in multiview video-plus-depth communications using geometric transforms

This paper addresses depth data recovery in multiview video-plus-depth communications affected by transmission errors and/or packet loss. The novel aspects of the proposed method rely on the use of geometric transforms and warping vectors, capable of capturing complex motion and view-dependent deformations, which are not efficiently handled by traditional motion and/or disparity compensation methods. By exploiting the geometric nature of depth information, a region matching approach combined with depth contour reconstruction is devised to achieve accurate interpolation of arbitrary shapes within lost regions of depth maps. The simulation results show that, for different packet loss rates, up to 20%, the depth maps recovered by the proposed method produce virtual views with better quality than existing methods based on motion information and spatial interpolation. An average PSNR gain of 1.48 dB is obtained in virtual views synthesised from depth maps using the proposed method.     

An adaptive quantization algorithm without side information for depth map coding

This paper presents a novel block-adaptive quantization scheme for efficient bit allocation without side information in depth map coding. Since the type of distortion in a depth map causes different effects in terms of the visual artifacts in a synthesized view, the proposed method adaptively assigns the number of bits according to the characteristics of the corresponding texture block. I have studied the details of the depth map and its rendered view distortion, modeled these analytically, and then proposed a new rate and distortion model for depth map coding. Finally, I derived a simple closed-form solution based on my proposed rate and distortion model, which determines the block-adaptive quantization parameter without any side information. Experimental results show that the proposed scheme can achieve coding gains of more than 0.6% and 1.4% for quarter- and full-resolution depth maps, respectively, in a multi-view-plus-depth 3D system.