网格运动分析与形态学滤波相结合的视频对象分割

提出了一种基于二维网格运动分析与改进形态学滤波空域自动分割策略相结合的视频对象时空分割算法。该算法首先利用高阶统计方法对视频图像的二维网格表示进行运动分析，快速得到前景对象区域，通过后处理有效获得前景对象运动检测掩膜。然后，用一种结合交变序列重建滤波算法和自适应阈值判别算法的改进分水岭分割策略有效获得前景对象的精确边缘。最后，用区域基时空融合算法将时域分割结果和空域分割结果结合起来提取出边缘精细的视频对象。实验结果表明，本算法综合了多种算法的优点，主客观分割效果理想。     

Two-dimensional mesh-based mosaic representation for manipulationof video objects with occlusion

We present a two-dimensional (2-D) mesh-based mosaic representation, consisting of an object mesh and a mosaic mesh for each frame and a final mosaic image, for video objects with mildly deformable motion in the presence of self and/or object-to-object (external) occlusion. Unlike classical mosaic representations where successive frames are registered using global motion models, we map the uncovered regions in the successive frames onto the mosaic reference frame using local affine models, i.e., those of the neighboring mesh patches. The proposed method to compute this mosaic representation is tightly coupled with an occlusion adaptive 2-D mesh tracking procedure, which consist of propagating the object mesh frame to frame, and updating of both object and mosaic meshes to optimize texture mapping from the mosaic to each instance of the object. The proposed representation has been applied to video object rendering and editing, including self transfiguration, synthetic transfiguration, and 2-D augmented reality in the presence of self and/or external occlusion. We also provide an algorithm to determine the minimum number of still views needed to reconstruct a replacement mosaic which is needed for synthetic transfiguration. Experimental results are provided to demonstrate both the 2-D mesh-based mosaic synthesis and two different video object editing applications on real video sequences.     

Video object segmentation in rainy situations based on difference scheme with object structure and color analysis

A scheme based on a difference scheme using object structures and color analysis is proposed for video object segmentation in rainy situations. Since shadows and color reflections on the wet ground pose problems for conventional video object segmentation, the proposed method combines the background construction-based video object segmentation and the foreground extraction-based video object segmentation where pixels in both the foreground and background from a video sequence are separated using histogram-based change detection from which the background can be constructed and detection of the initial moving object masks based on a frame difference mask and a background subtraction mask can be further used to obtain coarse object regions. Shadow regions and color-reflection regions on the wet ground are removed from the initial moving object masks via a diamond window mask and color analysis of the moving object. Finally, the boundary of the moving object is refined using connected component labeling and morphological operations. Experimental results show that the proposed method performs well for video object segmentation in rainy situations.     

VIDEO OBJECT SEGMENTATION BY 2-D MESH-BASED MOTION ANALYSIS

Video object extraction is a key technology in content-based video coding.A novel video object extracting algorithm by two Dimensional (2-D) mesh-based motion analysis is proposed in this paper.Firstly,a 2-D mesh fitting the original frame image is obtained via feature detection algorithm. Then,higher order statistics motion analysis is applied on the 2-D mesh representation to get an initial motion detection mask.After post-processing,the final segmenting mask is quickly obtained.And hence the video object is effectively extracted.Experimental results show that the proposed algorithm combines the merits of mesh-based segmenting algorithms and pixel-based segmenting algorithms,and hereby achieves satisfactory subjective and objective performance while dramatically increasing the segmenting speed.     

Occlusion-adaptive, content-based mesh design and forward tracking

Two-dimensional (2-D) mesh-based motion compensation preserves neighboring relations (through connectivity of the mesh) as well as allowing warping transformations between pairs of frames; thus, it effectively eliminates blocking artifacts that are common in motion compensation by block matching. However, available 2-D mesh models, whether uniform or non-uniform, enforce connectivity everywhere within a frame, which is clearly not suitable across occlusion boundaries. To this effect, we hereby propose an occlusion-adaptive forward-tracking mesh model, where connectivity of the mesh elements (patches) across covered and uncovered region boundaries are broken. This is achieved by allowing no node points within the background to be covered (BTBC) and refining the mesh structure within the model failure (MF) region(s) at each frame. The proposed content-based mesh structure enables better rendition of the motion (compared to a uniform or a hierarchical mesh), while tracking is necessary to avoid transmission of all node locations at each frame. Experimental results show successful motion compensation and tracking.     

Two-dimensional mesh-based visual-object representation forinteractive synthetic/natural digital video

This paper first provides an overview of two-dimensional (2-D) and three-dimensional mesh models for digital video processing. It then introduces 2-D mesh-based modeling of video objects as a compact representation of motion and shape for interactive, synthetic/natural video manipulation, compression, and indexing. The 2-D mesh representation and the mesh geometry and motion compression have been included in the visual tools of the upcoming MPEG-4 standard. Functionalities enabled by 2-D mesh-based visual-object representation include animation of still texture maps, transfiguration of video overlays, video morphing, and shape-and motion-based retrieval of video objects     

A novel moving object segmentation framework utilizing camera motion recognition for H.264 compressed videos

This paper presents a novel coarse to fine moving object segmentation framework for H.264/AVC compressed videos. The proposed framework integrates the global motion estimation and global motion compensation steps in the segmentation pipeline unlike previous techniques which did not consider such an integration. The integration is based on testing for presence of global motion by classifying the interframe motion vectors into moving camera class and still camera class. The decision boundary separating these two classes is learnt from the training video data. The integration automates the moving object segmentation to be applicable for static, moving and combination of static/moving camera cases which to the best of our knowledge has not been carried out earlier. Further, a novel coarse segmentation technique is proposed by decomposing the inter-frame motion vectors into wavelet sub-bands and utilizing logical operations on LH, HL and HH sub-band wavelet coefficients. The premise is based on the fact that since the LH, HL and HH sub-bands contain the detail information pertaining to horizontal, vertical and diagonal moving blocks respectively, they can be exploited to identify the coarse moving boundaries. The coarse segmentation is fast in comparison to state-of-the-art coarse segmentation methods as demonstrated by our experiments. Finally, these coarse boundaries are modeled in an energy minimization framework and shown that by minimizing the energy using graph cut optimization the segmentation is refined to obtain the fine segmentation. The proposed framework is tested on a number of standard video sequences encoded with H.264/AVC JM encoder and comparison is carried out with state-of-the-art compressed domain moving object segmentation methods as well as with an existing state-of-the-art pixel domain method to establish and validate the proposed moving object segmentation framework.     

基于模板匹配的视频对象分割

视频对象分割是MPEG-4标准关键技术.本文结合模板匹配和基于运动估值和补偿的对象跟踪方法,提出了一种可以从复杂场景中分割出MPEG-4视频对象的新方法.在使用运动估值和补偿得到分割掩膜后,以初始帧对象颜色为模板,在当前帧的轮廓边界区域通过模板匹配检测对象,使轮廓精确化.本文方法在一定范围内有效解决了遮挡问题,并能够以初始帧跟踪任意长序列中的对象.     

Improvement of region-based motion estimation by considering uncovered regions

In natural video sequences, object movement causes regions to be covered or uncovered. Conventional algorithms for region-based motion estimation do not take uncovered regions into full account. Uncovered regions seriously decrease the accuracy of motion estimation. This paper presents an algorithm for increasing the motion estimation accuracy. This algorithm detects uncovered regions and uses them to improve image segmentation and motion estimation. Experimental results show that the presented algorithm is effective in reducing the displaced frame difference, without introducing any extra information for coding applications.     

A novel recursive bayesian learning-based method for the efficient and accurate segmentation of video with dynamic background

Segmentation of video with dynamic background is an important research topic in image analysis and computer vision domains. In this paper, we present a novel recursive Bayesian learning-based method for the efficient and accurate segmentation of video with dynamic background. In the algorithm, each frame pixel is represented as the layered normal distributions which correspond to different background contents in the scene. The layers are associated with a confident term and only the layers satisfy the given confidence which will be updated via the recursive Bayesian estimation. This makes learning of background motion trajectories more accurate and efficient. To improve the segmentation quality, the coarse foreground is obtained via simple background subtraction first. Then, a local texture correlation operator is introduced to fill the vacancies and remove the fractional false foreground regions. Extensive experiments on a variety of public video datasets and comparisons with some classical and recent algorithms are used to demonstrate its improvements in both segmentation accuracy and efficiency.     

Multi-view video based multiple objects segmentation using graph cut and spatiotemporal projections

In this paper, we present an automatic algorithm to segment multiple objects from multi-view video. The Initial Interested Objects (IIOs) are automatically extracted in the key view of the initial frame based on the saliency model. Multiple objects segmentation is decomposed into several sub-segmentation problems, and solved by minimizing the energy function using binary label graph cut. In the proposed novel energy function, the color and depth cues are integrated with the data term, which is then modified with background penalty with occlusion reasoning. In the smoothness term, foreground contrast enhancement is developed to strengthen the moving objects boundary, and at the same time attenuates the background contrast. To segment the multi-view video, the coarse predictions of the other views and the successive frame are projected by pixel-based disparity and motion compensation, respectively, which exploits the inherent spatiotemporal consistency. Uncertain band along the object boundary is shaped based on activity measure and refined with graph cut, resulting in a more accurate Interested Objects (IOs) layer across all views of the frames. The experiments are implemented on a couple of multi-view videos with real and complex scenes. Excellent subjective results have shown the robustness and efficiency of the proposed algorithm.     

Contextual encoding in uniform and adaptive mesh-based lossless compression of MR images

We propose and evaluate a number of novel improvements to the mesh-based coding scheme for 3-D brain magnetic resonance images. This includes: 1) elimination of the clinically irrelevant background leading to meshing of only the brain part of the image; 2) content-based (adaptive) mesh generation using spatial edges and optical flow between two consecutive slices; 3) a simple solution for the aperture problem at the edges, where an accurate estimation of motion vectors is not possible; and 4) context-based entropy coding of the residues after motion compensation using affine transformations. We address only lossless coding of the images, and compare the performance of uniform and adaptive mesh-based schemes. The bit rates achieved (about 2 bits per voxel) by these schemes are comparable to those of the state-of-the-art three-dimensional (3-D) wavelet-based schemes. The mesh-based schemes have been shown to be effective for the compression of 3-D brain computed tomography data also. Adaptive mesh-based schemes perform marginally better than the uniform mesh-based methods, at the expense of increased complexity.     

Approach to automatic video motion segmentation

A novel, fast automatic motion segmentation approach is presented. It differs from conventional pixel or edge based motion segmentation approaches in that the proposed method uses labelled regions (facets) to segment various video objects from the background. Facets are clustered into objects based on their motion and proximity details using Bayesian logic. Because the number of facets is usually much lower than the number of edges and points, using facets can greatly reduce the computational complexity of motion segmentation. The proposed method can tackle efficiently the complexity of video object motion tracking, and offers potential for real-time content-based video annotation.     

背景渐变的视频对象分割算法研究及实现

提出了一种有效的背景渐变的视频对象分割算法.首先将前一帧分成前景和背景两部分,然后采用灰度投影匹配算法对当前帧进行全局运动估计和补偿,将当前帧与上一帧进行差分运算,便可得到差分图像.通过对差分图像进行二值化处理,得到运动模板并与前景信息进行相与计算,再结合当前帧信息便可得到运动目标.在TI公司的TMS320DM642芯片上验证了该算法,实验结果表明该算法不仅对亮度变化和环境变化具有鲁棒性,而且可独立、精确地分割出运动目标.     

Robust bilayer video segmentation by adaptive propagation of global shape and local appearance

Segmenting semantic objects of interest from video has long been an active research topic, with a wide range of potential applications. In this paper, we present a bilayer video segmentation method robust to abrupt motion and change in appearance for both the foreground and background. Specifically, based on a few manually segmented keyframes, the proposed method propagates the global shape of the foreground as priors to adjacent frames by applying branch-and-mincut [1], which jointly estimates what is optimal among a set of shapes along with its pose and the corresponding segmentation in the current image. Based on this preliminary segmentation we determine two types of local regions likely to have erroneous results, and apply a probabilistic framework where shape and appearance cues are adaptively emphasized for local refinement. With each successive frame segmentation, the set of shapes applied as priors are incrementally updated. Experimental results support the robustness of the proposed method for obstacles such as background clutter, motion, and appearance changes, from only a small number of user segmented keyframes.     

一种高精度的压缩域视频目标分割算法

该文提出了一种工作于MPEG压缩域的快速视频目标分割算法。该算法以从MPEG1/2码流中部分解码提取的特征为输入,提取P帧中的运动目标。针对一般的压缩域算法目标边界精度不高的特点,算法采用I帧和P帧中每个块的直流DCT系数和3个交流DCT系数,以及运动补偿信息,重建出P帧的原图像1/16大小的子图像,采用快速平均移聚类得到具有较高边界精度的亮度一致的区域;针对运动向量的噪声容易造成错误检测的缺点,算法结合聚类分析结果和运动块的分布,采用基于马尔可夫随机场的统计标号方法对目标和背景区域进行分类,得到每个P帧的目标掩模。该算法可以得到44子块的边界精度,对于CIF格式的码流,在Pentium IV 2GHz平台上可以达到每秒40帧的处理速度。     

Moving object detection and tracking from video captured by moving camera

This paper presents an effective method for the detection and tracking of multiple moving objects from a video sequence captured by a moving camera without additional sensors. Moving object detection is relatively difficult for video captured by a moving camera, since camera motion and object motion are mixed. In the proposed method, the feature points in the frames are found and then classified as belonging to foreground or background features. Next, moving object regions are obtained using an integration scheme based on foreground feature points and foreground regions, which are obtained using an image difference scheme. Then, a compensation scheme based on the motion history of the continuous motion contours obtained from three consecutive frames is applied to increase the regions of moving objects. Moving objects are detected using a refinement scheme and a minimum bounding box. Finally, moving object tracking is achieved using a Kalman filter based on the center of gravity of a moving object region in the minimum bounding box. Experimental results show that the proposed method has good performance.     

Robust global motion estimation oriented to video object segmentation.

Most global motion estimation (GME) methods are oriented to video coding while video object segmentation methods either assume no global motion (GM) or directly adopt a coding-oriented method to compensate for GM. This paper proposes a hierarchical differential GME method oriented to video object segmentation. A scheme which combines three-step search and motion parameters prediction is proposed for initial estimation to increase efficiency. A robust estimator that uses object information to reject outliers introduced by local motion is also proposed. For the first frame, when the object information is unavailable, a robust estimator is proposed which rejects outliers by examining their distribution in local neighborhoods of the error between the current and the motion-compensated previous frame. Subjective and objective results show that the proposed method is more robust, more oriented to video object segmentation, and faster than the referenced methods.     

一种从视频压缩码流中精确提取运动对象的快速算法

针对目前大部分视频对象分割方法相当复杂而且计算量大的问题,提出了一种在压缩域粗分割,在空域精细分割的方法。该方法利用压缩域中运动向量进行聚类,得到运动对象的初始分割。将分割模板通过运动参数映射到参考帧I帧,,解码初始分割区域进行Canny边缘俭测和边缘跟踪,即可得到精确的对象轮廓。该方法使得处理的数据量保持最小,节约了处理时间并得到了像素级精度的分割对象。     

Unsupervised video object segmentation with distractor-aware online adaptation

Unsupervised video object segmentation is a crucial application in video analysis when there is no prior information about the objects. It becomes tremendously challenging when multiple objects occur and interact in a video clip. In this paper, a novel unsupervised video object segmentation approach via distractor-aware online adaptation (DOA) is proposed. DOA models spatiotemporal consistency in video sequences by capturing background dependencies from adjacent frames. Instance proposals are generated by the instance segmentation network for each frame and they are grouped by motion information as positives or hard negatives. To adopt high-quality hard negatives, the block matching algorithm is then applied to preceding frames to track the associated hard negatives. General negatives are also introduced when there are no hard negatives in the sequence. The experimental results demonstrate these two kinds of negatives are complementary. Finally, we conduct DOA using positive, negative, and hard negative masks to update the foreground and background segmentation. The proposed approach achieves state-of-the-art results on two benchmark datasets, the DAVIS 2016 and the Freiburg-Berkeley motion segmentation (FBMS)-59.