Efficient Stereoscopic Video Coding Using Joint Disparity-Motion Estimation

The goal of this paper is to determine dense disparity and motion fields jointly and efficiently, given a sequence of stereoscopic images. At each time instant, the motion field of left sequences is estimated by using the current disparity field, the next disparity field, and the motion field of right sequences. The improved concepts for the proposed joint estimation are based on an edge-preserving regularization, an ambiguous vector region detection, and a median filtering for block-based vector generation. The reported approach is verified by processing a set of stereo sequences. Results are given with real stereoscopic data.     

一种基于立体视邻接帧时空相关性的最小代价函数帧估计算法

为了在立体视频序列编码中获得高的压缩率,需要对立体视频序列中一个视的序列按传统方法进行独立编码;另一个视的序列中,只对其中一些参考帧(I帧或P帧)按视差补偿预测的方法进行编码,其余帧不进行编码和传输,而在解码端用立体视帧估计的方法得到重建.本文提出了一种基于立体视中邻接帧在图像、视差场和运动矢量场之间高度相关性的方法.对于因遮挡而缺乏估计的区域,则结合了图像强度的连续性和运动,视差矢量的分布特性,构造了代价方程并估计出该部分的运动矢量及强度值.实验证明,重建出来的帧图像在视觉和信噪比意义上均具有较好的效果.     

3D reconstruction from stereo images for interactions between real and virtual objects

The 3D reconstruction algorithm in a stereo image pair for realizing mutual occlusion and interactions between the real and virtual world in an image synthesis is proposed. A two-stage algorithm, consisting of disparity estimation and regularization is used to locate a smooth and precise disparity vector. The hierarchical disparity estimation technique increases the efficiency and reliability of the estimation process, and edge-preserving disparity field regularization produces smooth disparity fields while preserving discontinuities that result from object boundaries. Depth information concerning the real scene is then recovered from the estimated disparity fields by stereo camera geometry. Simulation results show that the proposed algorithm provides accurate and spatially correlated disparity vector fields in various types of images, and the reconstructed 3D model produces a natural space in which the real world and virtual objects interact with each other as if they were in the same world.     

Joint depth-motion dense estimation for multiview video coding

The multiview video coding (MVC) extension of H.264/MPEG-4 AVC [1] is one of the most promising visual encoders for three-dimensional television and free viewpoint video applications. In this paper, we propose a joint dense motion/disparity estimation algorithm, designed to replace the classical temporal/inter-view unit within MVC, which uses a block-based motion/disparity estimation. The motion vector fields and the disparity vector fields are therefore simultaneously derived using the stereo-motion consistency constraint in a set theoretic convex optimization framework. The obtained displacement vector fields are then jointly segmented by minimizing a rate-distortion cost function, in line with the multiple reference frame strategy used in H.264/MPEG-4 AVC. Experimental results demonstrate the benefits of the proposed method compared to the separated dense estimation scheme or the block-based estimation technique.     

A joint encoder–decoder error control framework for stereoscopic video coding

Stereoscopic video coding (SSVC) plays an important role in various 3D video applications. In SSVC, robust stereoscopic video transmission over error-prone networks is still a challenge problem to be solved. In this paper, we propose a joint encoder–decoder error control framework for SSVC, where error-resilient source coding, transmission network conditions, and error concealment scheme are jointly considered to achieve better error robustness performance. The proposed joint encoder–decoder error control framework includes two parts: an error concealment algorithm at the decoder side and a rate–distortion optimized error resilience algorithm at the encoder side. For error concealment at the decoder side, an overlapped block motion and disparity compensation based error concealment scheme is proposed to adaptively utilize inter-view correlations and temporal correlations. For error resilience at the encoder side, first, the inter-view refreshment is proposed for SSVC to suppress error propagations. Then, an end-to-end distortion model for SSVC is derived, which jointly considers the transmission network conditions, inter-view refreshment, and error concealment tools at the decoder side. Finally, based on the derived end-to-end distortion model, the rate–distortion optimized error resilience algorithm is presented to adaptively select inter-view, inter- or intra-coding for SSVC. The experimental results show that the proposed joint encoder–decoder error control framework has superior error robustness performance for stereoscopic video transmission over error-prone networks.     

Disparity field and depth map coding for multiview 3D image generation

In the present paper techniques are examined for the coding of the depth map and disparity fields for stereo or multiview image communication applications. It is assumed that both the left and right channels of the multiview image sequence are coded using block- or object-based methods. A dynamic programming algorithm is used to estimate a disparity field between each stereo image pair. Depth is then estimated and occlusions are optionally detected, based on the estimated disparity fields. Spatial interpolation techniques are examined based on the disparity/depth information and the detection of occluded regions using either stereoscopic or trinocular camera configurations. It is seen that the presence of a third camera at the transmitter site improves the estimation of disparities, the detection of occlusions and the accuracy of the resulting spatial interpolation at the receiver. Various disparity field and depth map coding techniques are then proposed and evaluated, with emphasis given to the quality of the resulting intermediate images at the receiver site. Block-based and wireframe modeling techniques are examined for the coding of isolated depth or disparity map information. Further, 2D and 3D motion compensation techniques are evaluated for the coding of sequences of depth or disparity maps. The motion fields needed may be available as a byproduct of block-based or object-based coding of the intensity images. Experimental results are given for the evaluation of the performance of the proposed coding and spatial interpolation methods.     

Stereo video coding based on quad-tree decomposition of B-P frames by motion and disparity interpolation

A new optimised technique for coding stereoscopic image sequences is presented and compared with already known methods. The proposed technique, called enhanced interpolated motion and disparity estimation (EIMDE), is based on a joint method, which encodes the frames of the right image sequence by exploiting both the temporal redundancy of the same sequence and the disparity redundancy with the left image sequence. In the proposed method, a variable block size scheme has been employed for motion and disparity estimation. The block size is controlled by quad-tree decomposition of the processed frame based on a rate-distortion splitting criterion. For the prediction of a macroblock, optimised motion and disparity vectors are jointly estimated and the participating proportion of each similarity is suitably searched. In this way, the energy of the resulted residual frame is minimised and the whole framework is optimised. Finally, the residual frame is decomposed by a discrete wavelet transform and is further compressed by morphological encoding of the resulting coefficients. The proposed coder has been experimentally evaluated on real image sequences, where it produced good performance over other known methods.     

基于区域划分的立体视频整帧丢失错误隐藏新算法

针对立体视频传输中右视点整帧丢失,提出了一种基于区域划分的立体视频整帧丢失错误隐藏新算法。首先通过丢失帧视点内和视点间相邻帧的运动信息,估计丢失帧的运动区域与静止区域;然后利用视点间的全局视差,将丢失帧运动区域进一步划分为遮挡边界区域和非遮挡边界区域,并对遮挡边界区域和非遮挡边界区域分别采用运动补偿预测和视差补偿预测进行恢复,而对静止区域则直接采用帧拷贝的方法恢复丢失信息。实验结果表明,本文算法有效提高了所恢复的立体视频丢失帧图像的主客观质量。     

Stereo image analysis for multi-viewpoint telepresence applications

An improved method for combined motion and disparity estimation in stereo sequences to synthesize temporally and perspectively intermediate views is presented. The main problems of matching methods for motion and disparity analysis are summarised. The improved concept is based on a modified block matching algorithm in which a cost function consisting of feature- and area-based correlation together with an appropriately weighted temporal smoothness term is applied. Considerable improvements have been obtained with respect to the motion and disparity assignments by introducing a confidence measure to evaluate the reliability of estimated correspondences. In occluded image areas, enhanced results are obtained applying an edge-assisted vector interpolation strategy. Two different image synthesis concepts are presented. The first concept is suitable for processing natural stereo sequences. It comprises the detection of covered and uncovered image areas caused by motion or disparity. This information is used to switch between different interpolation and extrapolation modes during the computation of intermediate views. The proposed object-based approach is suitable for processing typical video conference scenes containing extremely large occluded image regions and keeping implementation costs low. A set of stereo sequences has been processed. The performed computer simulations show that a continuous motion parallax can be obtained with good image quality by using sequences taken with stereo cameras having large interaxial distances.     

A Stereoscopic Video Generation Method Using Stereoscopic Display Characterization and Motion Analysis

Stereoscopic video generation methods can produce stereoscopic content from conventional video filmed with monoscopic cameras. In this paper, we propose a stereoscopic video generation method using motion analysis which converts motion into disparity values and considers multi-user conditions and the characteristics of the display device. The field of view and the maximum and minimum disparity values were calculated in the stereoscopic display characterization stage and were then applied to various types of 3D displays. After motion estimation, we used three cues to decide the scale factor of motion-to-disparity conversion. These cues were the magnitude of motion, camera movements and scene complexity. A subjective evaluation showed that the proposed method generated more satisfactory video sequence.     

Extraction technique of region of interest from stereoscopic video

A feature fusion approach is presented to extract the region of interest (ROI) from the stereoscopic video. [0]Based on human vision system (HVS), the depth feature, the color feature and the motion feature are chosen as vision features. [0]The algorithm is shown as follows. Firstly, color saliency is calculated on superpixel scale. Color space distribution of the superpixel and the color difference between the superpixel and background pixel are used to describe color saliency and color salient region is detected. Then, the classic visual background extractor (Vibe) algorithm is improved from the update interval and update region of background model. The update interval is adjusted according to the image content. The update region is determined through non-obvious movement region and background point detection. So the motion region of stereoscopic video is extracted using improved Vibe algorithm. The depth salient region is detected by selecting the region with the highest gray value. Finally, three regions are fused into final ROI. Experiment results show that the proposed method can extract ROI from stereoscopic video effectively. In order to further verify the proposed method, stereoscopic video coding application is also carried out on the joint model (JM) encoder with different bit allocation in ROI and the background region.     

面向立体视频传输的右视点图像错误隐藏

利用立体视频序列视点间相关性及单视点内相关性,提出了一种面向立体视频传输的错误隐藏算法.从基于H.264/AVC立体视频编码结构出发,推断出受损块的参考模式;然后基于出错块的内容特征,根据块视差活力度(TDA)或块运动活力度(TMA)的大小,内容自适应地选择恰当的视点间及时域错误隐藏方法对受损块进行错误掩盖.实验结果表...     

基于均值漂移的立体视频对象分割

提出了一种结合深度信息与改进的均值漂移算法相结合的立体视频对象分割方法.首先采用自适应权值的视差估计方法.获得可靠的视差图,然后利用改进的均值漂移算法对视差图进行分割,获取视频对象.实验结果表明,该方法可以获得与语义一致的对象分割.     

自适应选择联合补偿预测模式的立体视频编码算法

运动补偿预测．视差补偿预测以及两者相结合的联合补偿预测是研究立体视频编码的关键技术。不同预测模式不仅考虑了左右视频通道各自的时间相关性、而且考虑到了左右视频通道之间的空间相关性。本文提出了一种自适应选择预测模式的立体视频编码算法,并在预测模式确定后通过选取不同的加权值来达到联合补偿预测的目的。实验结果表明,本文提出的算法在改善编码性能的同时,提高了编码速度。     

Edge-preserving disparity estimation and disparity-compensated intermediate view reconstruction for stereo images

A constrained disparity estimation method is proposed which uses a directional regularization technique to efficiently preserve edges for stereo image coding. The proposed method smoothes disparity vectors in smooth regions and preserves edges in object boundaries well, without creating an oversmoothing problem. The differential pulse code modulation (DPCM) technique for disparity map coding is used prior to entropy coding, in order to improve the overall coding efficiency. The proposed disparity estimation method can also be applied to intermediate view reconstruction. Intermediate views between a left image and a right image provide reality and natural motion parallax to multiviewers. Intermediate views are synthesized by appropriately exploiting an interpolation or an extrapolation technique according to the characteristics of each region after identifying the regions as occluded regions, normal regions, and regions having ambiguous disparities.The experimental results show that the proposed disparity estimation method gives close matches between a left image and a right image and improves coding efficiency. In addition, we can subjectively confirm that the application of our proposed intermediate view reconstruction method leads to satisfactory intermediate views from a stereo image pair.This work was supported by the Korea Institute of Science and Technology (KIST) under Grant No. 99HI-054.     

Super resolution image reconstruction through bregman iteration using morphologic regularization

Multiscale morphological operators are studied extensively in the literature for image processing and feature extraction purposes. In this paper, we model a nonlinear regularization method based on multiscale morphology for edge-preserving super resolution (SR) image reconstruction. We formulate SR image reconstruction as a deblurring problem and then solve the inverse problem using Bregman iterations. The proposed algorithm can suppress inherent noise generated during low-resolution image formation as well as during SR image estimation efficiently. Experimental results show the effectiveness of the proposed regularization and reconstruction method for SR image.     

基于一阶范数的视频超分辨率重建

讨论了视频成像的基本模型,并在此基础上提出了一种基于一阶范数的视频图像重建算法,该方法同时对高分辨率图像和运动矢量进行迭代估计,并采用一阶范数和Robers算子对于当前普遍的采用二阶范数项作为视频数据逼近项和归整项的方式进行了改进。实验结果表明,该算法在原图像受到椒盐噪声影响的情况下,重建效果要优于二阶方法,并且该方法在重建图象的边缘保持上也有相对比较好的效果。     

多视点视频编码中的视频目标提取与视差匹配

在单通道视频目标分割算法的基础上,首先提出一种多视点视频编码中的视频目标提取方案.然后提出一种基于多级块匹配视差估计算法.应用Moravec算子对视差矢量图进行检测,提取特征视差矢量并对其进行精确度检验.若特征视差矢量不够精确,则其所在块将重新进行小范围的块匹配.若块匹配误差仍太大则将其分裂为四个子块重新匹配.与传统块匹配算法相比,新方法能获得较为精确和平滑的视差场,提高重建后的图像质量.     

一种基于MPEG-2的立体视频编码中的视差匹配快速算法

高效，快速的视差匹配是立体视频处理中的一项关键技术。本文在分析立体图像序列的视差矢量与运动矢量之间的相关性的基础上，提出一种基于MPEG-2的立体视频编码中的视差匹配快速算法。实验结果表明，与全搜索法相比，在保证重建图像质量的前提下，快速算法能显著降低视差估计的计算复杂度。