Bilayer segmentation of webcam videos using tree-based classifiers

This paper presents an automatic segmentation algorithm for video frames captured by a (monocular) webcam that closely approximates depth segmentation from a stereo camera. The frames are segmented into foreground and background layers that comprise a subject (participant) and other objects and individuals. The algorithm produces correct segmentations even in the presence of large background motion with a nearly stationary foreground. This research makes three key contributions: First, we introduce a novel motion representation, referred to as "motons," inspired by research in object recognition. Second, we propose estimating the segmentation likelihood from the spatial context of motion. The estimation is efficiently learned by random forests. Third, we introduce a general taxonomy of tree-based classifiers that facilitates both theoretical and experimental comparisons of several known classification algorithms and generates new ones. In our bilayer segmentation algorithm, diverse visual cues such as motion, motion context, color, contrast, and spatial priors are fused by means of a conditional random field (CRF) model. Segmentation is then achieved by binary min-cut. Experiments on many sequences of our videochat application demonstrate that our algorithm, which requires no initialization, is effective in a variety of scenes, and the segmentation results are comparable to those obtained by stereo systems.     

Fractional Stereo Matching Using Expectation-Maximization

In our fractional stereo matching problem, a foreground object with a fractional boundary is blended with a background scene using unknown transparencies. Due to the spatially varying disparities in different layers, one foreground pixel may be blended with different background pixels in stereo images, making the color constancy commonly assumed in traditional stereo matching not hold any more. To tackle this problem, in this paper, we introduce a probabilistic framework constraining the matching of pixel colors, disparities, and alpha values in different layers, and propose an automatic optimization method to solve a maximizing a posterior (MAP) problem using expectation-maximization (EM), given only a short-baseline stereo input image pair. Our method encodes the effect of background occlusion by layer blending without requiring a special detection process. The alpha computation process in our unified framework can be regarded as a new approach by natural image matting, which handles appropriately the situation when the background color is similar to that of the foreground object. We demonstrate the efficacy of our method by experimenting with challenging stereo images and making comparisons with state-of-the-art methods.     

视频特技中色键效果算法

针对视频编辑软件中的色键效果算法,从理论上进行了深入研究。通过 数学建模,得到了求解前景中视频对象的蒙片方程和通式,在分析特例的基础上,提出了一种双前景抠象技术,并给出了相应的定理及证明,利用它可将任意颜色的视频对象从任意底色中抠出,并通过C语言编程实现了该技术。     

Fast interactive stereo image segmentation

The paper presents an approach to cutting out the same target object from a pair of stereo images interactively. With this approach, a user labels parts of the object and background in either of the images with strokes. The approach generates a segmentation result immediately. In case it is not satisfying, the result can be improved by interactively drawing more strokes, or using an alternative interaction way called adding corresponding points, which is first presented in this paper. The proposed segmentation approach is capable of providing feedback fast after each interaction. The fast computation is performed in the framework of graph cut. First, the labeled parts are used to learn foreground and background color models. Next, an energy function is built by formulating the similarities between unlabeled pixels and the foreground/background color models, color difference between neighbor pixels, and stereo correspondences obtained by SIFT feature matching. At last, graph cut is utilized to find the optimum of the energy function and obtain a segmentation result. Different from state-of-the-art methods, our segmentation approach formulates sparse correspondences rather than dense matches as stereo constraints in the energy function. Experimental results demonstrate that our method is faster in computation. In the meanwhile, it generates comparable results with state-of-the-art methods.     

一种将羽毛球比赛的2D视频转换到3D视频的算法

文中提出一种羽毛球比赛的2D视频转换到3D视频的算法。在这类视频中,前景是最受关注的部分,准确地从背景中提取出前景对象是获取深度图的关键。文中采用一种改进的图割算法来获取前景,并根据场景结构构建背景深度模型,获取背景深度图;在背景深度图的基础上,根据前景与镜头之间的距离关系为前景对象进行深度赋值,从而得到前景深度图。然后,融合背景深度图和前景深度图,得到完整的深度图。最后,通过基于深度图像的虚拟视点绘制技术DIBR来获取用于3D显示的立体图像对。实验结果表明,最终生成的立体图像对具有较好的3D效果。     

Stereo matching with linear superposition of layers

In this paper, we address stereo matching in the presence of a class of non-Lambertian effects, where image formation can be modeled as the additive superposition of layers at different depths. The presence of such effects makes it impossible for traditional stereo vision algorithms to recover depths using direct color matching-based methods. We develop several techniques to estimate both depths and colors of the component layers. Depth hypotheses are enumerated in pairs, one from each layer, in a nested plane sweep. For each pair of depth hypotheses, matching is accomplished using spatial-temporal differencing. We then use graph cut optimization to solve for the depths of both layers. This is followed by an iterative color update algorithm which we proved to be convergent. Our algorithm recovers depth and color estimates for both synthetic and real image sequences.     

基于模糊判别的快速图切割立体匹配

针对图切割算法存在匹配时间过长的不足,提出了一种快速的图切割立体匹配方法.选用模糊逻辑提取少量的候选匹配点,然后用这些候选匹配点构造缩减的图,再进行图切割立体匹配.采用标准图像对文中方法进行测试,并与图切割立体算法和动态规划算法进行比较的结果表明,该方法较好地保留了图切割算法匹配的准确性,而运行时间仅为图切割算法的1/35.     

基于稀疏卷积的前景实时双目深度估计算法

为了提高立体匹配算法处理前景视差估计任务的计算效率,针对一般网络采用完全双目图像作为输入,场景内前景空间占比小而导致输入信息冗余度大的缺点,提出了一种基于稀疏卷积的目标实时立体匹配算法。为实现和改善算法对稀疏前景的视差估计,首先利用分割算法同时获得稀疏前景掩膜和场景语义特征;其次通过稀疏卷积提取稀疏前景区域的空间特征后与场景语义特征相融合,并将融合特征输入到解码模块进行视差回归;最后以前景真值图作为损失生成视差图。在ApolloScape数据集上的测试结果表明,所提算法的准确性和实时性均优于先进算法PSMNet和GANet,且算法的单次运行时间低至60.5 ms,对目标遮挡具有一定的鲁棒性,可用于目标实时深度估计。     

结合全卷积网络的无监督视频目标分割

对视频中的目标进行像素级分割是计算机视觉领域的研究热点，完全没有用户标注的无监督视频分割对分割算法提出了更高的要求。近几年在分割中常使用基于帧间运动信息进行建模的方法，即用光流等运动信息预测目标轮廓，再结合颜色等特征建立模型进行分割。针对这些方法产生的前景背景混淆以及边缘粗糙等问题，本文提出结合全卷积网络的视频目标分割方法。首先通过全卷积网络预测视频序列中显著目标的轮廓，结合光流获得的运动显著性标签进行修正，然后建立时间-空间图模型，运用图割的方法获得最终的预测标签。在SegTrack v2以及DAVIS这2个通用数据集上进行评估，结果表明本文方法较基于帧间运动信息的方法在分割效果上有明显的提高。     

一种基于Boosting判别模型的运动阴影检测方法

在视频处理中,由于运动阴影具有与运动前景相同的特性,当在提取前景时,会误把阴影检测为前景.特别是当阴影和其它前景发生粘连时,这可能会严重地影响跟踪、识别等后续处理.该文提出了一种用于运动阴影检测的Boosting判别模型.这种方法先利用Boosting在不同的特征空间来区分前景和阴影,然后在判别随机场(DRFs)中结合前景和阴影的时空一致性,实现对前景和阴影的分割.首先,差分前图像与背景图像得到颜色不变子空间和纹理不变子空间;然后在这两个子空间上应用Boosting来区分前景和阴影;最后利用前景和阴影的时空一致性,在判别随机场中通过图分割的方法准确地分割前景和阴影.实验结果表明,无论是在室内场景,还是在室外场景,该文的方法要好于传统的方法.     

结合置信度加权融合与视觉注意机制的前景检测

目的 在视频前景检测中,像素级的背景减除法检测结果轮廓清晰,灵活性高。然而,基于样本一致性的像素级分类方法不能有效利用像素信息,遇到颜色伪装和出现静止前景等复杂情形时无法有效检测前景。为解决这一问题,提出一种基于置信度加权融合和视觉注意的前景检测方法。方法 通过加权融合样本的颜色置信度和纹理置信度之和判断前景,进行自适应更新样本的置信度和权值;通过划分子序列结合颜色显著性和纹理差异度构建视觉注意机制判定静止前景目标,使用更新置信度最小样本的策略保持背景模型的动态更新。结果 本文方法在CDW2014（change detection workshops 2014）和SBM-RGBD（scene background modeling red-green-blue-depth）数据集上进行检测,相较于5种主流算法,本文算法的查全率和精度相较于次好算法分别提高2.66%和1.48%,综合性能最优。结论 本文算法提高了在颜色伪装和存在静止前景等复杂情形下前景检测的精度和召回率,在公开数据集上得到更好的检测效果。可将其应用于存在颜色伪装和静止前景等复杂情形的视频监控中。     

Video fingerprinting based on graph model

A robust video fingerprinting based on graph model is proposed in this paper, where two graph models are constructed for key frames selection and foreground extraction, respectively. First, the video is represented as a complete undirected graph and a binary tree is formed using normalized cut algorithm to select key frames. Then, the pixels of each key frame are modeled as a Markov Random Field and another graph model is formed to extract foreground by graph cut. Finally, the fourth-order cumulant of foreground is computed to generate video fingerprints. Experimental results show that the proposed algorithm has good robustness and discrimination.     

消除光照影响的背景减除算法

为了准确分离图像中的对象与背景信息,并使得分离效果不受图像中光照变化的影响,提出了一种改进的图像背景减除算法。首先,算法使用改进的图割方法进行背景减除,提高减除效果。其次,算法提出颜色衰减模型和对比度衰减模型,改进能量公式中的颜色分量和对比度分量,消除光照变化对于减除结果的影响。最后,针对分离出的二值图像,提出利用形态学操作和空洞填充策略予以优化补偿,优化最终图像分离的结果。实验结果表明,当原始图像和背景图像光线强度差别较大,或光线被对象遮挡的情况下,相比于传统背景减除算法,该算法的减除效果仍然较好。该算法适用于复杂环境下图像中对象提取。     

Multiple piecewise constant with geodesic active contours (MPC-GAC) framework for interactive image segmentation using graph cut optimization

This paper proposes an improved variational model, multiple piecewise constant with geodesic active contour (MPC-GAC) model, which generalizes the region-based active contour model by Chan and Vese, 2001 [11] and merges the edge-based active contour by Caselles et al., 1997 [7] to inherit the advantages of region-based and edge-based image segmentation models. We show that the new MPC-GAC energy functional can be iteratively minimized by graph cut algorithms with high computational efficiency compared with the level set framework. This iterative algorithm alternates between the piecewise constant functional learning and the foreground and background updating so that the energy value gradually decreases to the minimum of the energy functional. The k-means method is used to compute the piecewise constant values of the foreground and background of image. We use a graph cut method to detect and update the foreground and background. Numerical experiments show that the proposed interactive segmentation method based on the MPC-GAC model by graph cut optimization can effectively segment images with inhomogeneous objects and background.     

一种沿区域边界的动态规划立体匹配算法

提出一种基于图像区域分割的立体匹配算法.与通常的沿扫描行进行动态规划的立体匹配算法不同,该算法以图像"区域"为基元计算视差.首先使用相关法得到初始视差,然后利用一种区域边界上的多种子动态规划算法对视差进行精细计算,最终通过插值得到整个图像的稠密视差.实验结果表明,此算法速度较快、可靠性较高.     

Video chroma keying via global sampling and trimap propagation

Chroma keying is a widely used video editing technique, which finely separates the foreground objects from the background. Two major concerns are involved in chroma keying problems: alpha estimation and foreground color restoration. The alpha values reveal the opacity property of the foreground objects. The foreground color restoration removes the background color influence to the foreground appearance especially at transparent regions and objects’ boundaries. In this paper, the color range of the solid background is well analyzed to automatically separate foreground from background. Global sampling is utilized to robustly and reliably estimate the foreground color at boundaries and transparent regions. Furthermore, we propose to propagate the geometric shape of foreground boundaries between adjacent frames by using optical flow and thin plate splines interpolation. The trimap, which is an initial foreground/background/unknown segmentation of each frame can be automatically updated for each video frame by using our proposed propagation method. Compared to previous methods, our proposed matting method estimates high-quality alpha matte and reliable foreground color with least user interference.     

Efficient Dense Stereo with Occlusions for New View-Synthesis by Four-State Dynamic Programming

A new algorithm is proposed for efficient stereo and novel view synthesis. Given the video streams acquired by two synchronized cameras the proposed algorithm synthesises images from a virtual camera in arbitrary position near the physical cameras. The new technique is based on an improved, dynamic-programming, stereo algorithm for efficient novel view generation. The two main contributions of this paper are: (i) a new four state matching graph for dense stereo dynamic programming, that supports accurate occlusion labelling; (ii) a compact geometric derivation for novel view synthesis by direct projection of the minimum cost surface. Furthermore, the paper presents an algorithm for the temporal maintenance of a background model to enhance the rendering of occlusions and reduce temporal artefacts (flicker); and a cost aggregation algorithm that acts directly in the three-dimensional matching cost space. The proposed algorithm has been designed to work with input images with large disparity range, a common practical situation. The enhanced occlusion handling capabilities of the new dynamic programming algorithm are evaluated against those of the most powerful state-of-the-art dynamic programming and graph-cut techniques. Four-state DP is also evaluated against the disparity-based Middlebury error metrics and its performance found to be amongst the best of the efficient algorithms. A number of examples demonstrate the robustness of four-state DP to artefacts in stereo video streams. This includes demonstrations of cyclopean view synthesis in extended conversational sequences, synthesis from a freely translating virtual camera and, finally, basic 3D scene editing.     

Stereo Matching with Transparency and Matting

This paper formulates and solves a new variant of the stereo correspondence problem: simultaneously recovering the disparities, true colors, and opacities of visible surface elements. This problem arises in newer applications of stereo reconstruction, such as view interpolation and the layering of real imagery with synthetic graphics for special effects and virtual studio applications. While this problem is intrinsically more difficult than traditional stereo correspondence, where only the disparities are being recovered, it provides a principled way of dealing with commonly occurring problems such as occlusions and the handling of mixed (foreground/background) pixels near depth discontinuities. It also provides a novel means for separating foreground and background objects (matting), without the use of a special blue screen. We formulate the problem as the recovery of colors and opacities in a generalized 3D (x, y, d) disparity space, and solve the problem using a combination of initial evidence aggregation followed by iterative energy minimization.