面向裸眼立体显示终端的立体视频编码方法

X3D裸眼可视立体显示器需要同时输入8个视点的信息,用户在一定角度范围内可以看到不同视角的立体画面显示.面向裸眼立体显示终端提出了一种立体视频编码方法,在编码端完成复杂的视差估计,压缩后的视差信息通过网络传输到立体显示终端以降低系统对用户终端的要求.用户终端采用基于视差图绘制的方法快速绘制出6个中间视点,实现裸眼可视立体显示器上的多视角立体画面显示.针对视差图的处理,给出了全局平滑和非边缘区域平滑两种视差图的处理方法.实验结果表明,与不对视差图进行平滑的方法相比,对视差图进行平滑能有效提高编码的整体率失真性能.     

Correspondence

Abstract

According to the Nyquist sampling theorem, a large number of sampled images and small intervals between capturing cameras should be met for rendering high quality virtual views without aliasing, which is hard to realize in practice. Therefore, achieving a balance between multi-view data capturing and quality of the rendered view remains as open problems. To solve this problem, we analysed the spectral bounds of the scene and designed a reconstruction filter. A proper number for rendering and a three-dimensional surface describing the relation between multi-view data capturing and quality of the rendered view were derived. Experimental results for both the modelled scene and the real scene show that only about 20% of sample images are needed compared with Nyquist sampling, while the quality of the rendered view remains higher than that of a Nyquist sampled comparison.     

Virtual view synthesis method and self‐evaluation metrics for free viewpoint television and 3D video

Virtual view synthesis is one of the most important techniques to realize free viewpoint television and three‐dimensional (3D) video. In this article, we propose a view synthesis method to generate high‐quality intermediate views in such applications and new evaluation metrics named as spatial peak signal‐to‐noise ratio and temporal peak signal‐to‐noise ratio to measure spatial and temporal consistency, respectively. The proposed view synthesis method consists of five major steps: depth preprocessing, depth‐based 3D warping, depth‐based histogram matching, base plus assistant view blending, and depth‐based hole‐filling. The efficiency of the proposed view synthesis method has been verified by evaluating the quality of synthesized images with various metrics such as peak signal‐to‐noise ratio, structural similarity, discrete cosine transform (DCT)‐based video quality metric, and the newly proposed metrics. We have also confirmed that the synthesized images are objectively and subjectively natural. © 2010 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 20, 378–390, 2010     

一种基于多边形柱面全景图的虚拟漫游新方法

提出了一种基于图像绘制的多边形柱面全景图的虚拟漫游方法。利用普通的手持相机在一个多边形区域内沿某一路径拍摄并拼接多幅全景图,通过基于SIFT的特征点检测来计算深度,用狭缝图像插值来实现整个区域内的平滑漫游。该方法具有采样简单、虚拟场景真实感强,支持连续大范围漫游的特点。     

3D model-based pose invariant face recognition from multiple views

A 3D model-based pose invariant face recognition method that can recognise a human face from its multiple views is proposed. First, pose estimation and 3D face model adaptation are achieved by means of a three-layer linear iterative process. Frontal view face images are synthesised using the estimated 3D models and poses. Then the discriminant `waveletfaces' are extracted from these synthesised frontal view images. Finally, corresponding nearest feature space classifier is implemented. Experimental results show that the proposed method can recognise faces under variable poses with good accuracy     

A 3D modeling and free‐view generation system using environmental stereo cameras

We propose a 3D video system that uses environmental stereo cameras to display a target object from an arbitrary viewpoint. This system is composed of the following stages: image acquisition, foreground segmentation, depth field estimation, 3D modeling from depth and shape information, and arbitrary view rendering. To create 3D models from captured 2D image pairs, a real‐time segmentation algorithm, a fast depth reconstruction algorithm, and a simple and efficient shape reconstruction method were developed. For viewpoint generation, the 3D surface model is rotated toward the desired place and orientation, and the texture data extracted from the original camera is projected onto this surface. Finally, a real‐time system that demonstrates the use of the aforementioned algorithms was implemented. The generated 3D object can easily be manipulated, e.g., rotated or translated, to render images from different viewpoints. This provides stable scenes of a minimal area that made it possible to understand the target space, and also made it easier for viewers to understand in near real‐time. © 2008 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 17, 367–378, 2007     

Research on adaptive scanning and quantisation for 3D DCT video coding

Abstract

Nowadays, a well-established video coding method is based on the block-matching algorithm that is in the core of all MPEG and H.26x standards. However, this method involves motion estimation and compensation; thus the computational complexity is high. However, the video coding based on three-dimensional discrete cosine transform (3D DCT) is a potential method, and the scanning order and quantisation of 3D DCT coefficients play a crucial role in coding effect. So, the statistical performances of 3D DCT are studied in this paper, and then adaptive scanning order and quantisation of 3D coefficients are proposed. The theoretical analysis and experiment results show significant improvement in performance over previously reported methods.     

Novel fast block motion estimation using diamond‐arc‐hexagon search patterns

Abstract

This work presents a novel block motion estimation (ME) approach, diamond‐arc‐hexagon search (DAHS), which uses diamond, arc, and hexagon search patterns to accomplish the fast searching process. According to the characteristic of the center‐based motion vector distribution for the real‐world video sequence, DAHS employs a small diamond pattern extended hexagon search to refine the central area search. Furthermore for estimation in large motion situations, the strategy of the large diamond‐to‐arc search can preserve good performance. Experimental results indicate that DAHS is suitable for both (quasi‐) stationary and large motion searches. DAHS outperforms the efficient three‐step (E3SS) and hexagon‐based search (HEXBS) in prediction quality, and improves the searching speed about 9% compared to E3SS. The proposed algorithm was tested using several sequences and delivered excellent performance in search speed and prediction of video quality. The DAHS can be applied to real‐time video processes, such as H.264/MPEG‐4 advanced video coding (AVC) and scalable video coding (SVC).     

6D vector orthogonal transformation and its application in multiview video coding

Abstract

Multiview video (MVV) is multiple video sequences that integrated different viewpoints data of the same three-dimensional (3D) scene. Each viewpoint data are taken from the ordinary video camera. Thus, the data are very large for the MVV. So compression is necessary in order to store and transmit effectively. Based on the theory of multi-dimensional vector matrix (MDVM), we propose a six-dimensional (6D) vector orthogonal transform nuclear matrix, and prove its orthogonality and energy concentration. We apply the theory to multiview video coding (MVC). This transformation is based on discrete cosine transform (DCT), which has the optimal performance for video data. We represent MVV data with a multi-dimensional (MD) mathematical model. The chosen MVV is earlier eight frames in YUV format from two viewpoints. We divide the Y, U and V components into cubes respectively, and combine the two views data into one cube, on which the transformation is conducted. Good results are obtained in terms of energy concentration. This paper provides a new method for handling MVV, and prepare for the next quantisation and coding.     

Intra mode decision and frame layer Lagrange multiplier adjustment scheme for H.264 based on structural similarity

H.264 takes rate distortion optimisation (RDO) technique to perform intra and inter mode decision and achieves higher coding efficiency, but the objective distortion metric such as mean square error (MSE) is employed in traditional RDO framework, which cannot acquire optimal subjective quality. In this paper, structural similarity (SSIM)-based subjective distortion is applied to RDO-based intra mode decision in H.264 I frame video coding, and a linear SSIM distortion model is firstly proposed and SSIM-based rate distortion cost function for intra mode decision is defined. Furthermore, a content adaptive frame layer Lagrange multiplier adjustment scheme is proposed to balance the tradeoff between rate and SSIM distortion better. Experimental results show that, the proposed method encodes image structural information more effectively and thus acquires better perceptual quality and subjective RDO performance compared with objective distortion-based RDO method. Under the same perceptual quality, our scheme achieves about 8·03% I frame bit rate reduction on average for various sequences over MSE-based RDO employed in JM reference software.     

一种新的可伸缩小波视频编码方法

提出了一种高度可伸缩的视频编码系统,首先对图象序列进行时间、水平和垂直方面的三维小波分解,并采用改进的集合分区等级树算法（ＳＰＩＨＴ）对小波系数进行量化。在量化过程,对压缩比特流进行排序和重组,以便中根据需要从中抽取某些子集进行译码。本编码方案帧率、空间分辨率和失真水平等的可伸缩性,对于在移动通信信道及ＡＴＭ网中进行视频传输业务是非常有用的。     

Analysis and correction of ultrasonic wavefront distortion based on a multilayer phase-screen model

A model is introduced that incorporates the cumulative wavefront distortion effects caused by spatial heterogeneities along the path of propagation, and a corresponding model-based wavefront distortion-correction method is presented. In the proposed model, a distributed heterogeneous medium is lumped into a series of parallel phase screens. The distortion effects can be compensated-without a priori knowledge of the distorting structure-by backpropagation of received wavefronts through hypothetical multiple phase screens located between the imaging system and targets, while each pointwise time shift is adjusted iteratively to maximize a specified image quality factor at the final layer. Theoretical analyses indicate that the mean speckle brightness decreases monotonically with the root-mean-square value of distributed phase distortions; therefore, the speckle brightness can be used as an image quality factor. Experimental one-dimensional (1-D) array data with simulated distortion effects based on a real 2-D abdominal-tissue map were used to evaluate the performance of the proposed method and existing aberration-correction techniques. The simulated characteristics of wavefront distortion and relative performance of existing correction techniques were similar to reports based on abdominal-wall data and breast data. This investigation shows that the proposed method provides better compensation for wavefront distortion.     

视频失真校准方法的研究

针对视频失真量的校准需求,分析了视频微分增益失真dG和微分相位失真dP的测量方法和基本原理,采用数字信号合成方法开发出标准视频失真信号序列,建立了视频失真信号源标准装置,详细阐述了其校准系统和测量不确定度,将视频信号失真量溯源至电压标准和相位标准。     

Hierarchical prediction structure for subimage coding and multithreaded parallel implementation in integral imaging

We are concerned with the coding of subimage-transformed elemental images to solve the problems of data transmission and storage in three-dimensional (3D) integral imaging in this paper. First, we use the subimage transform for preprocessing of the elemental image array (EIA). Because of the similarity of correlation distributions between the subimage array (SIA) and multiview video, we present a hierarchical prediction structure for SIA coding based on the hierarchical B picture (HBP) structure for multiview video coding. Moreover, we design a multithreaded parallel implementation for the proposed structure according to inter-row prediction dependencies. Experiments are performed on both EIAs and SIAs. The results show that employing the same coding strategy, the proposed parallel implemented HBP scheme achieves not only higher image quality and better 3D effect but also lower coding delay at low bit rates compared with the previously reported Hilbert-curve-based scheme.     

全局运动估计及其在视频编码中的应用

对全局运动估计进行了研究,并提出了它的改进算法。在有局部运动的情况下,改进算法的计算速度和准确性都有明显提高。研究了全局运动估计在视频编码中的应用。在视频压缩它是spirte编码的关键技术,这 种技术较之传统编码方法有较好的编码性能。利用MPEG－4的软件测试平台进行了MPEG－4校验模型中已有算法的实验比较,结果表明,改进的全局运动估计的计算速度明显提高,视频编码的性能得到了改善。     

基于HVS和时域感知失真的感兴趣区域搜索

视频质量评价模型对于感知视频编码有着重要意义.观察者对于视频场景中不同区域有不同的视觉兴趣性,而度量不同区域人眼感兴趣程度,对于构建高性能视频质量评价模型非常重要.我们在研究中发现,时域失真和时域波动分布是影响视觉兴趣性最重要的特征因素,因此定量度量视频序列的时域感知失真和时域失真波动,并根据这两个参量,采用自适应阈值判断的算法,标定出可能的兴趣像素点;同时对选中的可能兴趣点,进行空域连通分析.根据连通区域面积大小,确定1～5个可能的感兴趣区域.并根据聚类算法,确定最终的区域.最后结果证明了算法的可行性.     

Estimating three-dimensional vehicle motion in an outdoor scene using stereo image sequences

We propose a motion estimation system that uses stereo image pairs as the input data. To perform experimental work, we also obtain a sequence of outdoor stereo images taken by two metric cameras. The system consists of four main stages, which are (1) determination of point correspondences on the stereo images, (2) correction of distortions in image coordinates, (3) derivation of 3D point coordinates from 2D correspondences, and (4) estimation of motion parameters based on 3D point correspondences. For the first stage of the system, we use a four-way matching algorithm to obtain matched point on two stereo image pairs at two consecutive time instants (t_i and t_{i + 1}). Since the input data are stereo images taken by cameras, it has two types of distortions, which are (i) film distortion and (ii) lens distortion. These two distortions must be corrected before any process can be applied on the matched points. To accomplish this goal, we use (i) bilinear transform for film distortion correction and (ii) lens formulas for lens distortion correction. After correcting the distortions, the results are 2D coordinates of each matched point that can be used to derive 3D coordinates. However, due to data noise, the calculated 3D coordinates to not usually represent a consistent rigid structure that is suitable for motion estimation; therefore, we suggest a procedure to select good 3D point sets as the input for motion estimation. The procedure exploits two constraints, rigidity between different time instants and uniform point distribution across the object on the image. For the last stage, we use an algorithm to estimate the motion parameters. We also wish to know what is the effect of quantization error on the estimated results; therefore an error analysis based on quantization error is performed on the estimated motion parameters. In order to test our system, eight sets of stereo image pairs are extracted from an outdoor stereo image sequence and used as the input data. The experimental results indicate that the proposed system does provide reasonable estimated motion parameters.     

Highly efficient CAVLC encoder for MPEG-4 AVC/H.264

In dealing with high-resolution video information, encoding (or decoding) with an efficient context-based adaptive variable length code (CAVLC) encoder is important. A highly efficient CAVLC encoder is proposed for video coding application of MPEG-4 AVC/H.264. The main concept is to use block-based pipelining to speed up encoding efficiency and reduce the pipeline storage elements by using the associated input buffer. We also use zero-block detection to speed up encoding efficiency and eliminate the same codeword from all the tables to save the hardware cost. Simulation results show that our design can meet the real-time processing for 1920 times 1088 resolution with lower operation frequency. We also accomplish the higher encoding throughput with a more complete CAVLC design than others. The proposed design has been implemented and synthesised with TSMC 0.18 mum standard cell library. The synthesis result indicates that the gate count is 12 125 with the clock constraint of 125 MHz.     

Optimization of wireframe model adaptation and motion estimation in a rate-distortion framework

A rate-distortion framework is used to define a very low-bit-rate coding scheme based on wireframe model adaptation and optimized selection of motion estimators. This technique achieves maximum reconstructed image quality under the constraint of a target bit rate for the coding of the vector field and the wireframe representation information. First, a complete scheme is proposed for hybrid two-dimensional (2D) and 3D motion estimation and compensation. The wireframe adaptation and updating is optimized for hybrid motion estimation in the rate distortion sense. A more sophisticated technique, adapted to the requirements of a very low-bit-rate coder is also proposed which considers also the transmission of the prediction error corresponding to the particular choice of the motion estimator for each object in the scene. Experimental results illustrating the performance of the proposed techniques in very low-bit-rate image sequence coding application areas are presented and evaluated. © 1998 John Wiley & Sons, Inc. Int J Imaging Syst Technol, 9, 238–247, 1998     

A method for correcting radial distortion based on verifying the planarity of specimens

Image distortion is inevitable when an image is captured through a lens. While the digital image measurement technique is getting popular, image distortion problem can result in significant error. A new distortion correction method is proposed in this study. The proposed method is based on the fact that a flat surface should keep flat when it is measured using three-dimensional (3D) digital image measurement technique. The 3D digital image measurement technique adopted in this research is the simplified 3D digital image correlation (DIC) method. Because radial distortion has a more noticeable influence than other types of distortions, this method deals only with radial distortion. A few experiments are carried out in this study to verify the correctness of this method and its accuracy. Both simulated data and actual image data are adopted in these experiments. The results show that this method can achieve a good accuracy. The standard deviations caused by random errors are about the same order as the random errors. It also shows that this method is suitable for both large and small distortion conditions.