Disparity field and depth map coding for multiview 3D image generation

In the present paper techniques are examined for the coding of the depth map and disparity fields for stereo or multiview image communication applications. It is assumed that both the left and right channels of the multiview image sequence are coded using block- or object-based methods. A dynamic programming algorithm is used to estimate a disparity field between each stereo image pair. Depth is then estimated and occlusions are optionally detected, based on the estimated disparity fields. Spatial interpolation techniques are examined based on the disparity/depth information and the detection of occluded regions using either stereoscopic or trinocular camera configurations. It is seen that the presence of a third camera at the transmitter site improves the estimation of disparities, the detection of occlusions and the accuracy of the resulting spatial interpolation at the receiver. Various disparity field and depth map coding techniques are then proposed and evaluated, with emphasis given to the quality of the resulting intermediate images at the receiver site. Block-based and wireframe modeling techniques are examined for the coding of isolated depth or disparity map information. Further, 2D and 3D motion compensation techniques are evaluated for the coding of sequences of depth or disparity maps. The motion fields needed may be available as a byproduct of block-based or object-based coding of the intensity images. Experimental results are given for the evaluation of the performance of the proposed coding and spatial interpolation methods.     

Artifact Trapping During Time Reversal Photoacoustic Imaging for Acoustically Heterogeneous Media

Several different reconstruction algorithms have been proposed for photoacoustic tomography, most of which presuppose that the acoustic properties of the medium are constant and homogeneous. In practice, there are often unknown spatial variations in the acoustic properties, and these algorithms give, at best, only approximate estimates of the true image. The question as to which approach is the most robust in these circumstances is therefore one of practical importance. Image reconstruction by “time reversal”—using a numerical propagation model with a time-varying boundary condition corresponding to the measured data in reversed temporal order—has been shown to be less restrictive in its assumptions than most, and therefore a good candidate for a general and practically useful algorithm. Here, it is shown that such reconstruction algorithms can “trap” time reversed scattered waves, leading to artifacts within the image region. Two ways to mitigate this effect are proposed.      

Noise Reduction in Image Sequences Using Motion-Compensated Temporal Filtering

Noise in television signals degrades both the image quality and the performance of image coding algorithms. This paper describes a nonlinear temporal filtering algorithm using motion compensation for reducing noise in image sequences. A specific implementation for NTSC composite television signals is described, and simulation results on several video sequences are presented. This approach is shown to be successful in improving image quality and also improving the efficiency of subsequent image coding operations.     

Transform coding of stereo image residuals

Stereo image compression is of growing interest because of new display technologies and the needs of telepresence systems. Compared to monoscopic image compression, stereo image compression has received much less attention. A variety of algorithms have appeared in the literature that make use of the cross-view redundancy in the stereo pair. Many of these use the framework of disparity-compensated residual coding, but concentrate on the disparity compensation process rather than the post compensation coding process. This paper studies specialized coding methods for the residual image produced by disparity compensation. The algorithms make use of theoretically expected and experimentally observed characteristics of the disparity-compensated stereo residual to select transforms and quantization methods. Performance is evaluated on mean squared error (MSE) and a stereo-unique metric based on image registration. Exploiting the directional characteristics in a discrete cosine transform (DCT) framework provides its best performance below 0.75 b/pixel for 8-b gray-scale imagery and below 2 b/pixel for 24-b color imagery, In the wavelet algorithm, roughly a 50% reduction in bit rate is possible by encoding only the vertical channel, where much of the stereo information is contained. The proposed algorithms do not incur substantial computational burden beyond that needed for any disparity-compensated residual algorithm.     

Block motion compensated coding of interlaced sequences using adaptively deinterlaced fields

Video coding algorithms using block motion compensation were first developed for progressively scanned sequences and as such, are not entirely suitable for interlaced sequences In this paper we present a new approach for block-based coding of interlaced sequences. This proposed algorithm processes the interlaced sequence as a sequence of even and odd fields by using the last decoded field, adaptively deinterlaced, for the motion compensated prediction of the current field. The deinterlacing is performed at the decoder and no extra information has to be sent to guide the adaptation. The algorithm is a simple and efficient alternative to algorithms using the last two decoded fields for the motion compensated prediction of the current field. The new approach can easily incorporate the use of fast search algorithms and allows the use of true half-pixel accuracy in the estimates of the vertical component of the motion vectors. In HDTV sequences tested, this algorithm achieves superior performance due to this half-pixel accuracy.     

On the adaptive three-dimensional transform coding techniques for moving images

In this paper, an adaptive three-dimensional transform coding technique based on the 3-D discrete cosine transform (DCT) for removing the temporal correlation is proposed. Because of the nonstationary nature of the image data, the energy distribution in a 3-D DCT block varies along the vertical, horizontal and temporal directions. Thus, adaptive schemes, such as the 3-D classification, the classified linear scanning technique and the VLC table selection scheme, are used to take local variations into account. Also, in our approach, a hybrid technique, which adaptively combines relatively simple inter-frame coding with intra-frame coding, is presented. Through intensive computer simulations, the performance of the proposed 3-D transform coding technique is evaluated on several well-known moving sequences. The results show that, especially for moving sequences containing slow or moderate motion, the proposed technique provides an improved performance over the scheme with motion compensation (CCITT, 1989) at rates above 0.5 b/pixel (bpp), and a good visual quality of the reconstructed images is also obtained. Thus, the proposed 3-D transform coding technique is believed to be a good candidate for the digital VCR, since motion compensation is not required in the proposed 3-D coding technique.     

基于运动和视差信息的立体视频质量客观评价

在研究人类 立体视觉特性及现有立体图像/视频质量评价算法的基础上,提出了一种基于运动信息和视 差信息的立 体视频质量的客观评价方法。方法包括视频质量评价(VQA)和视频立体感评价(VSSA)两个指 标,其中VQA的估计基于梯度的结构相似度(GSSIM) 算法,并充分考虑了帧内的亮度信息和结构信息、帧间运动信息以及人眼的感知特性对视频 质量的影响, 特别是根据人类的视觉特性,对左右视点的质量赋予了不同的权重;VSSA的估计 是通过计算参考 视频的绝对差值图和降质视频的绝对差值图之间的峰值信噪比(PSNR)而得到。实验结果表明,本文方法对基于H.264 编码的失真视频的评价结果与主观测试有较高的一致性,很好地体现人眼的视觉特性。     

Edge oriented block motion estimation for video coding

Intensity-based block motion estimation and compensation algorithms are widely used to exploit temporal redundancies in video coding, although they suffer from several drawbacks. One of the problems is that blocks located on boundaries of moving objects are not estimated accurately. It causes poor motion-compensated prediction along the moving edges to which the human visual system is very sensitive. By considering the characteristics of block motions for typical image sequences, an intelligent classifier is proposed to separate blocks containing moving edges to improve on conventional intensity-based block matching approaches. The motion vectors of these blocks are computed using edge matching techniques, so that the motion-compensated frames are tied more closely to the physical features. The proposed method can then make use of this accurate motion information for edge blocks to compute the remaining non-edged blocks. Consequently, a fast and efficient block motion estimation algorithm is developed. Experimental results show that this approach gives a significant improvement in accuracy for motion-compensated frames and computational complexity, in comparison with the traditional intensity-based block motion estimation methods     

Attentive transmission

A simple attention-based model is proposed for efficient transmission of visual information using multiresolution structures. Images are sampled nonuniformly in space and time, such that sampling is dense at the focus of attention and sparse in the periphery (retinal-like). Assuming that the focus of attention usually corresponds to eye position while scanning an image, image features which are “eye catching” (such as sharp edges, motion, and high flicker rate) are used to drive the dense center of sampling. The transmitted image is reconstructed by combining each new sample with previous samplings to give progressive transmission. The selection of sampling points depends only on previously transmitted information, and only sampled values without their location need to be transmitted.     

Backward motion compensation for interlaced HDTV

The pel-recursive approach to motion estimation has been widely studied for compensating progressively scanned, moderate-resolution video. Although pel-recursive algorithms may not be suitable for application to interlaced high-definition television (HDTV), the underlying principle of backward motion compensation, upon which pel-recursive algorithms are based, can be exploited to improve the existing motion compensation algorithms. This paper proposes applying a backward approach to motion compensation to improve the performance of standard block-based algorithms for motion-compensated interlaced HDTV sequences. First, we describe a framework for motion compensation in which motion information is parameterized by a motion operator and a domain for that operator. Within this framework, we characterize the type of motion information represented by forward (e.g. block-based) and backward approaches to motion compensation. We propose a method for combining these two sources of motion information to form an optimal motion-compensated prediction. Simulations on two interlaced HDTV sequences demonstrate performance improvements between 1 and 2 dB over standard block-based methods.     

Foveation scalable video coding with automatic fixation selection

Image and video coding is an optimization problem. A successful image and video coding algorithm delivers a good tradeoff between visual quality and other coding performance measures, such as compression, complexity, scalability, robustness, and security. In this paper, we follow two recent trends in image and video coding research. One is to incorporate human visual system (HVS) models to improve the current state-of-the-art of image and video coding algorithms by better exploiting the properties of the intended receiver. The other is to design rate scalable image and video codecs, which allow the extraction of coded visual information at continuously varying bit rates from a single compressed bitstream. Specifically, we propose a foveation scalable video coding (FSVC) algorithm which supplies good quality-compression performance as well as effective rate scalability. The key idea is to organize the encoded bitstream to provide the best decoded video at an arbitrary bit rate in terms of foveated visual quality measurement. A foveation-based HVS model plays an important role in the algorithm. The algorithm is adaptable to different applications, such as knowledge-based video coding and video communications over time-varying, multiuser and interactive networks.     

3D motion estimation for depth information compression in 3D-TV applications

A new approach for predicting and coding depth information in 3D-TV (three-dimensional television) applications is presented. Properties of the depth information, which complements monoscopic video and enables 3D experience, are used in the proposed 3D motion prediction. The new approach leads to more efficient motion compensation and finally to higher compression. Built on the top of the conventional approaches for video coding, the proposed technique is suitable for integration in upcoming 3D-TV products.     

Frequency domain analysis of rotational motion

The consideration of translational motion in the frequency domain has rendered valuable service in many applications, such as television signal analysis, motion estimation, image registration, and visual perception studies. This work presents rotational motion in the frequency domain, developing theoretical results parallel to those relative to translations. The analysis can support the extension of frequency —domain—based approaches to specific applications concerning rotations, an operation successfully carried out by recent image registration works.     

Very low bit-rate coding algorithm for stereo video withspatiotemporal HVS model and binary correlation disparity estimator

A strategy for efficiently coding stereo video sequences is investigated. To fully utilize the suppression and the contrast sensitivity property of the human visual system, a novel coding scheme with two special mechanisms, the spatiotemporal HVS model and the binary correlation disparity estimator, is proposed to efficiently reduce the video signal redundancy and the computational complexity, while maintaining a high subjective image quality. Compared with existing stereo video coding systems, the proposed coding scheme supports a lower transmission bit rate and has less computational complexity. The simulation results also show that the subjective image quality of the reconstructed full color stereo sequences at 0.25-0.4 bits per pixel (bpp) is satisfactory     

Stereo image analysis for multi-viewpoint telepresence applications

An improved method for combined motion and disparity estimation in stereo sequences to synthesize temporally and perspectively intermediate views is presented. The main problems of matching methods for motion and disparity analysis are summarised. The improved concept is based on a modified block matching algorithm in which a cost function consisting of feature- and area-based correlation together with an appropriately weighted temporal smoothness term is applied. Considerable improvements have been obtained with respect to the motion and disparity assignments by introducing a confidence measure to evaluate the reliability of estimated correspondences. In occluded image areas, enhanced results are obtained applying an edge-assisted vector interpolation strategy. Two different image synthesis concepts are presented. The first concept is suitable for processing natural stereo sequences. It comprises the detection of covered and uncovered image areas caused by motion or disparity. This information is used to switch between different interpolation and extrapolation modes during the computation of intermediate views. The proposed object-based approach is suitable for processing typical video conference scenes containing extremely large occluded image regions and keeping implementation costs low. A set of stereo sequences has been processed. The performed computer simulations show that a continuous motion parallax can be obtained with good image quality by using sequences taken with stereo cameras having large interaxial distances.     

Global point tracking based panoramic image stabilization system

A novel image stabilization system is presented,which consists of a global feature point tracking based motion estimation,a Kalman filtering based motion smoothing and an image mosaic based panoramic compensation.The global motion is estimated using feature point matching and iteration with the least-square method.Then,the Kalman filter is applied to smooth the original motion vectors to effectively alleviate unwanted camera vibrations and follow the intentional camera scan.Lastly,the loss information of im...     

Invertible temporal subband/wavelet filter banks with half-pixel-accurate motion compensation

Three-dimensional (3-D) subband/wavelet coding with motion compensation has been demonstrated to be an efficient technique for video coding applications in some recent research works. When motion compensation is performed with half-pixel accuracy, images need to be interpolated in both temporal subband analysis and synthesis stages. The resulting subband filter banks developed in these former algorithms were not invertible due to image interpolation. In this paper, an invertible temporal analysis/synthesis system with half-pixel-accurate motion compensation is presented. We look at temporal decomposition of image sequences as a kind of down-conversion of the sampling lattices. The earlier motion-compensated (MC) interlaced/progressive scan conversion scheme is extended for temporal subband analysis/synthesis. The proposed subband/wavelet filter banks allow perfect reconstruction of the decomposed video signal while retaining high energy compaction of subband transforms. The invertible filter banks are then utilized in our 3-D subband video coder. This video coding system does not contain the temporal DPCM loop employed in the conventional hybrid coder and the earlier MC 3-D subband coders. The experimental results show a significant PSNR improvement by the proposed method. The generalization of our algorithm for MC temporal filtering at arbitrary subpixel accuracy is also discussed.     

Three-dimensional parameter estimation from stereo image sequences for model-based image coding

It is shown that the analysis of moving image sequences for 3D modelling can be performed in a relatively straightforward manner if the scene is captured in stereo. Output from a stereo disparity estimation process using calibrated cameras gives absolute 3D surface coordinates from a single stereo pair. When combined with monocular motion cues, the true 3D motion parameters of moving objects can be accurately calculated. Further analysis enables segmentation of body elements according to motion while the 3D surface feature structure, although available from the start, can be integrated and checked for anomalies over the sequence. These results are expected to alleviate the known problems of ambiguity suffered by monocular-source model-based coders.     

Image segmentation through graph-based clustering from surface normals estimated by photometric stereo

A method for segmenting 2D images based on 3D shape information is proposed. First, a robust photometric stereo technique estimates the 3D normals of the objects present in the scene for every image pixel. Then, the image is segmented by grouping its pixels according to their estimated normals through graph-based clustering. Differently from other image segmentation algorithms based on intensity, colour or texture, the regions of which are determined by the visual appearance of the depicted objects, the regions obtained with the proposed technique only depend on the 3D shapes of those objects. This can be advantageous for higher level scene understanding algorithms. This technique is especially suited to poorly illuminated scenarios and utilises a conventional camera and six inexpensive strobe lights.