Temporal stabilization of Video Object Segmentation for 3D-TV applications

Our aim is to insert depth information into an existing 2D video sequence to provide content for 3D-TV applications, which we try to achieve through segmentation of the objects in the given 2D video sequence. To this effect, we present a method for temporal stabilization of video object segmentation algorithms for 3D-TV applications. First, two quantitative measures to evaluate temporal stability without ground-truth are discussed. Then, a pseudo-3D curve evolution method, which spatio-temporally stabilizes the estimated segmentation of a video object is introduced. Temporal stability is achieved by re-distributing existing object segmentation errors such that they will be less disturbing when the scene is rendered and viewed in 3D. Our starting point is the hypothesis that if making segmentation errors is inevitable, these errors should be made in a temporally consistent way for 3D-TV applications. This hypothesis is supported by the experiments, which show that there is significant improvement in segmentation quality both in terms of the objective quantitative measures and in terms of the viewing comfort in subjective perceptual tests. Therefore, it is possible to increase the perceptual object segmentation quality without increasing the actual segmentation accuracy.     

新一代基于HEVC的3D视频编码技术

HEVC标准出台后,新一代基于HEVC的多视点加深度编码也将正式推出。基于HEVC的3D视频编码作为HEVC标准的扩展部分,主要面向立体电视和自由立体视频。从该编码方式的基本结构出发,较全面地介绍了视频编码方式、深度图编码方式和对深度图的编码控制三个方面的关键技术,包括视点间运动预测、深度图建模模式和视点合成优化等技术。     

面向三维高效视频编码的深度图错误隐藏

基于多视点视频序列视点内、视点间存在的相关性,并结合视点间运动矢量共享技术,该文提出一种面向3维高效视频编码中深度序列传输丢包的错误隐藏算法。首先,根据3D高效视频编码(3D-HEVC)的分层B帧预测(HBP)结构和深度图纹理特征,将深度图丢失块分成运动块和静止块;然后,对于受损运动块,使用结合纹理结构的外边界匹配准则来选择相对最优的运动/视差矢量进行基于位移矢量补偿的错误掩盖,而对受损静止块采用参考帧直接拷贝进行快速错误隐藏;最后,使用参考帧拆分重组来获取新的运动/视差补偿块对修复质量较差的重建块进行质量提升。实验结果表明:相较于近年提出的对比算法,该文算法隐藏后的深度帧平均峰值信噪比(PSNR)能提升0.25～2.03 dB,结构相似度测量值(SSIM)能提升0.001～0.006,且修复区域的主观视觉质量与原始深度图更接近。     

Depth-based direct mode for multiview video coding

Multiview video plus depth sequence is considered as an efficient 3D video format for supporting advanced stereoscopic and auto-stereoscopic multiview displays. In order to encode this video format, several modes are commonly employed with rate distortion optimization technique. Specifically, direct mode is an efficient mode to encode homogeneous or stationary regions without encoding any additional motion information. However, in the format, the conventional direct mode causes inaccuracy of motion parameters, i.e., a motion vector and a reference index. Thus, we propose a new method for deriving motion parameters of direct mode, which settles the inaccuracy problem, based on depth information and priority of neighboring blocks determined by their relative position. The proposed direct mode achieves an improvement in coding performance when compared with two state-of-the-art direct modes for multiview video coding. The experimental results show that the proposed method yields average bitrate savings of 4.6%, up to 13.7% in terms of the Bjontegaard Delta metric.     

High-performance depth map coding for 3D-AVC

Achieving high-performance coding for a depth map is one of the most important challenges in 3D video coding. In this paper, a quality enhancement method is proposed to accomplish better coding efficiency. A new technique separating contour and flat regions is designed, and a contour-aware quality enhancement algorithm is presented to improve depth map quality. We also propose a fast mode decision process to reduce computational complexity. The proposed fast algorithm uses similarity between texture video and depth map coding. The encoding process for a depth map is terminated early by using coded information from a texture video. Experimental results show that the quality of the depth map is improved by 0.11–0.59 dB, which translates into a bit rate saving of 2.19–8.19 %. The proposed fast algorithm saves encoding time, on average, by 36.4 %.     

一种基于运动补偿三维小波的 多描述视频编码方法

 本文将多描述编码与运动补偿三维小波可扩展视频编码相结合,提出了一种基于运动补偿三维小波的多描述视频编码方法.该方法首先根据编码序列的运动特性,自适应地进行每个描述的码率分配,以控制各个描述中的冗余,然后将编码序列的关键信息-运动矢量和低频帧码流复制到两个描述中,并将高频帧码流分配到不同的描述中.在解码端根据正确接收信息的不同,采用不同的方法进行视频重建.实验结果表明,与单描述编码方法相比,在信道丢包率较高的情况下,本文方法可以提供更好的传输鲁棒性.     

Fast depth map mode decision based on depth–texture correlation and edge classification for 3D-HEVC

The 3D extension of High Efficiency Video Coding (3D-HEVC) has been adopted as the emerging 3D video coding standard to support the multi-view video plus depth map (MVD) compression. In the joint model of 3D-HEVC design, the exhaustive mode decision is required to be checked all the possible prediction modes and coding levels to find the one with least rate distortion cost in depth map coding. Furthermore, new coding tools (such as depth-modeling mode (DMM) and segment-wise depth coding (SDC)) are exploited for the characteristics of depth map to improve the coding efficiency. These achieve the highest possible coding efficiency to code depth map, but also bring a significant computational complexity which limits 3D-HEVC from real-time applications. In this paper, we propose a fast depth map mode decision algorithm for 3D-HEVC by jointly using the correlation of depth map-texture video and the edge information of depth map. Since the depth map and texture video represent the same scene at the same time instant (they have the same motion characteristics), it is not efficient to use all the prediction modes and coding levels in depth map coding. Therefore, we can skip some specific prediction modes and depth coding levels rarely used in corresponding texture video. Meanwhile, the depth map is mainly characterized by sharp object edges and large areas of nearly constant regions. By fully exploiting these characteristics, we can skip some prediction modes which are rarely used in homogeneity regions based on the edge classification. Experimental results show that the proposed algorithm achieves considerable encoding time saving while maintaining almost the same rate-distortion (RD) performance as the original 3D-HEVC encoder.     

View upsampling optimization for mixed resolution 3D video coding

3D video is composed out of two or more, temporally synchronized, 2D video streams acquired at different camera poses and accompanied by geometrical information. In a mixed resolution 3D video stream, a subset of views is coded at reduced resolution. It has been shown in the literature that subjective quality of mixed resolution 3D video is close to that of full resolution 3D video. In order to improve the coding gain in mixed resolution coding scenario we present a new depth encoding method called view upsampling optimization. A novel depth distortion metric based on the performance of the depth-based super resolution is also presented. Finally, to improve the quality of the decoded video an improved depth-based super resolution method that uses view synthesis quality mapping is used for upsampling of low resolution views. The simulations, performed with the recently standardized MVC+D encoder, show that the proposed solution combined with the state of the art view synthesis distortion outperforms the anchor MVC+D coding scheme by 14.5 % of dBR on average for the total coded bitrate and by 17 % of dBR on average for the synthesized views.     

Discontinuity-based registration of depth and video data in depth image based rendering

Depth image based rendering (DIBR) has been proposed to create content for 3D-TV. In DIBR, stereoscopic images are created from monoscopic images and associated depth data. Since for most of the available video content sensor depth data are lacking, methods to create artificial depth data for video content have been developed. Yet artificial as well as sensor depth data may contain misalignments with respect to video data. Misaligned depth data are a source of artifacts observable in rendered 3D views. We show that by using an edge-based registration method, the spatial alignment of depth and video data can be improved, leading to an alleviation of the observed artifacts.     

基于多视点视频特征的3D-HEVC深度视频信息隐藏算法

为了更安全地传输秘密信息和保护3D视频,提 出基于多视点视频特征的3D-HEVC深度视频信息隐 藏算法。首先结合彩色视频的纹理特征,考虑深度视频不同区域对绘制视点的质量和编码效 率的影响,对 深度视频进行区域分割。然后针对不同的区域,采用不同的方式调制最大编码单元(LCU) 的QP值嵌入 秘密信息。最后,用修改后的QP值进行编码压缩,传输视频信息。实验仿真结果表明,相 比于原始HTM13.0, 本算法编码重建深度视频绘制视点质量的PSNR平均下降0.0015dB, 码率平均增加0.035%。本文算法 能较好地保证绘制视点的质量,对视频流的码率影响较小,且能实现秘密信息的盲提取。     

Early Merge mode decision algorithm for 3D-HEVC based on learning model

3D-high efficiency video coding (3D-HEVC) standard is an extension of HEVC.Though 3D-HEVC effectively improves the compression efficiency of 3D video,it also brings huge computational complexity.To greatly reduce the 3D-HEVC coding complexity,an early Merge mode decision approach was proposed.The residual signal that encoded by the Merge mode was firstly extracted as feature information.A learning model was established in terms of the residual signals of the coding unit (CU) in current frame that used early Merge mode as the optimal mode.Finally,the residual signal was extracted for the Merge mode of current CU,and the learning model was used to predict whether the Merge mode was the optimal mode or not.Experimental results show that the proposed early Merge mode decision approach reduces the coding times of 3D-HEVC texture views and depth maps about 41.9% and 24.3%,respectively,and the coding performance degradation is almost negligible.Compared with existing early Merge mode decision approaches,the proposed approach further reduces the coding time,and can be easily integrated into the 3D-HEVC test model due to its design simplicity.     

3D视频生成与编码的时间一致性深度图估计

In this paper, we propose a new algorithm for temporally consistent depth map estimation to generate three-dimensional video. The proposed algorithm adaptively computes the matching cost using a temporal weighting function, which is obtained by block-based moving object detection and motion estima-tion with variable block sizes. Experimental results show that the proposed algorithm im-proves the temporal consistency of the depth video and reduces by about 38% both the flic-kering artefact in the synthesized view and the number of coding bits for depth video coding.     

抗重压缩编码的3D-HEVC视频零水印算法

考虑到零水印算法的优越性以及目前针对3D高效视频编码标准的视频水印算法少的情况,提出了一种抗重压缩编码的视频零水印算法。首先,利用I帧的深度图及非I帧的运动矢量和编码单元划分情况对水印的构造位置进行选择。其次,通过全相位双正交变换及奇异值分解对16x16块进行处理,获取最大奇异值的最高有效位作为最终的特征信息。最后,得到的特征信息与图像水印进行异或生成零水印并进行零水印注册。实验结果表明,在不同量化参数的重压缩编码攻击、基于帧及重压缩编码混合攻击和信号处理及重压缩编码混合攻击下,本文算法具有良好的鲁棒性。      

Real-time generation of multi-view video plus depth content using mixed narrow and wide baseline

Content production for stereoscopic 3D-TV displays has become mature in the past years while huge progress has also been achieved in the improvement of the image quality of glasses-free auto-stereoscopic displays and light-field displays. Concerning the latter two display families, the content production workflow is less elaborated and more complex, as the number of required views not only differs considerably but is also likely to increase in the near future. As a co-existence of all 3D display families can be expected for the next years, one aims to establish an efficient content production workflow which yields to high quality content for all 3D-TV displays.Against this background we present a real-time capable multi-view video plus depth (MVD) content production workflow based on a four-camera rig with mixed narrow and wide baseline. Results show the suitability of the approach to simultaneously produce high quality MVD4 and native stereoscopic 3D content.     

Cross-layer optimized authentication and error control for wireless 3D medical video streaming over LTE

3D video for tele-medicine applications is gradually gaining momentum since the 3D technology can provide precise location information. However, the weak link for 3D video streaming is the necessary wireless link of the communication system. Neglecting the wireless impairments can severely degrade the performance of 3D video streaming that communicates complex critical medical data. In this paper, we propose systematic methodology for ensuring high performance of the 3D medical video streaming system. First, we present a recursive end-to-end distortion estimation approach for MVC (multiview video coding)-based 3D video streaming over error-prone networks by considering the 3D inter-view prediction. Then, based on the previous model, we develop a cross-layer optimization scheme that considers the LTE wireless physical layer (PHY). In this optimization, the authentication requirements of 3D medical video are also taken into account. The proposed cross-layer optimization approach jointly controls and manages the authentication, video coding quantization of 3D video, and the modulation and channel coding scheme (MCS) of the LTE wireless PHY to minimize the end-to-end video distortion. Experimental results show that the proposed approach can provide superior 3D medical video streaming performance in terms of peak signal-to-noise ratio (PSNR) when compared to state-of-the-art approaches that include joint source-channel optimized streaming with multi-path hash-chaining based-authentication, and also conventional video streaming with single path hash-chaining-based authentication.     

A novel multi-view image coding scheme based on view-warping and 3D-DCT

Efficient compression of multi-view images and videos is an open and interesting research issue that has been attracting the attention of both academic and industrial world during the last years. The considerable amount of information produced by multi-camera acquisition systems requires effective coding algorithms in order to reduce the transmitted data while granting good visual quality in the reconstructed sequence. The classical approach of multi-view coding is based on an extension of the H.264/AVC standard, still based on motion prediction techniques. In this paper we present a novel approach that tries to fully exploit the redundancy between different views of the same scene considering both texture and geometry information. The proposed scheme replaces the motion prediction stage with a 3D warping procedure based on depth information. After the warping step, a joint 3D-DCT encoding of all the warped views is provided, taking advantage of the strong correlation among them. Finally, the transformed coefficients are conveniently quantized and entropy coded. Occluded regions are also taken into account with ad-hoc interpolation and coding strategies. Experimental results performed with a preliminary version of the proposed approach show that at low bitrates it outperforms the H.264 MVC coding scheme on both real and synthetic datasets. Performance at high bitrates are also satisfactory provided that accurate depth information is available.     

3D-HEVC中深度图编码技术研究进展

3D-HEVC是为了满足3D视频和自由视点视频的高效编码而最新制定的视频编码标准,它要求同时编码几个视点的纹理视频和深度图.完全采用传统的技术来编码深度图会使得深度图内部锐利边界处产生伪影效应,为此,一些新的针对于深度图的编码工具被开发.详细介绍了这些编码工具,同时介绍了编码深度图时所使用的率失真优化方法.     

Unified depth intra coding for 3D video extension of HEVC

With the development of high-efficiency video coding (HEVC), the newest video coding standard, 3D video extension of HEVC (3D-HEVC) has been actively investigated. Since 3D-HEVC uses multi-view texture and depth data for input, various coding tools have been added to HEVC. In 3D-HEVC, on top of the existing 35 HEVC intra modes, eight additional modes exist, which are specifically for depth coding. In this paper, we propose a unified depth intra coding method that incorporates such depth intra modes into the regular intra mode set. In particular, the most rarely used HEVC intra modes are replaced by depth intra modes. As a result, binarization for depth intra modes is removed. Furthermore, the most probable mode selection procedure is modified to consider the elimination of several angular intra modes. The proposed method is implemented and tested on 3D video HEVC test model version 7.0. Simulation results report 2.2 % synthesis gain under all-intra configuration.     

Adaptive depth truncation filter for MVC based compressed depth image

In multiview video plus depth (MVD) format, virtual views are generated from decoded texture videos with corresponding decoded depth images through depth image based rendering (DIBR). 3DV-ATM is a reference model for the H.264/AVC based multiview video coding (MVC) and aims at achieving high coding efficiency for 3D video in MVD format. Depth images are first downsampled then coded by 3DV-ATM. However, sharp object boundary characteristic of depth images does not well match with the transform coding based nature of H.264/AVC in 3DV-ATM. Depth boundaries are often blurred with ringing artifacts in the decoded depth images that result in noticeable artifacts in synthesized virtual views. This paper presents a low complexity adaptive depth truncation filter to recover the sharp object boundaries of the depth images using adaptive block repositioning and expansion for increasing the depth values refinement accuracy. This new approach is very efficient and can avoid false depth boundary refinement when block boundaries lie around the depth edge regions and ensure sufficient information within the processing block for depth layers classification. Experimental results demonstrate that the sharp depth edges can be recovered using the proposed filter and boundary artifacts in the synthesized views can be removed. The proposed method can provide improvement up to 3.25 dB in the depth map enhancement and bitrate reduction of 3.06% in the synthesized views.     

多视点纹理加深度编码的联合码率控制方法

码率控制技术是多视点视频编码和传输中一个关键的问题。为了提高三维(3D)视频的整体显示质量,包括虚拟视点质量和编码视点质量,提出一种多视点纹理加深度编码的联合码率控制方法。该算法研究了纹理和深度的关系,采用基于模型方法确定最优的纹理和深度之间的码率比例。根据各个视点编码结果的统计规律,不同的视频序列采用不同的视点间比特分配比例。实验结果表明,与目前流行的多视点码率控制算法相比,该算法在计算复杂度基本保持不变的情况下,平均码率控制误差在0.6%以内,客观质量PSNR最高可提高0.65 dB。