基于时间一致性局部颜色特征的无纹理3D物体实时跟踪

针对无纹理3D物体跟踪算法在复杂背景和运动模糊的情况下容易跟踪失败、跟踪速度难以达到强实时等问题,提出一种基于时间一致性局部颜色特征的3D物体实时跟踪算法.首先在物体3D模型投影轮廓法向搜索线上计算像素颜色的加权均值作为局部颜色特征,增强颜色特征在复杂环境中的表征能力,并对局部颜色特征进行时间一致性更新,剔除前景背景颜色相似的局部颜色特征,以避免相似前景背景颜色导致的跟踪失败;然后定义基于局部颜色特征的能量函数,并推导该能量函数的解析导函数;最后改进了优化物体姿态的高斯牛顿法,通过添加阻尼参数防止姿态优化陷入局部极值,提高姿态估计精度和跟踪速度.实验使用7组测试视频验证文中算法,结果表明,该算法能更有效地克服复杂背景和运动模糊的干扰,在未使用并行计算的前提下可实现强实时跟踪.     

Tracking objects with generic calibrated sensors: An algorithm based on color and 3D shape features

We present a color and shape based 3D tracking system suited to a large class of vision sensors. The method is applicable, in principle, to any known calibrated projection model. The tracking architecture is based on particle filtering methods where each particle represents the 3D state of the object, rather than its state in the image, therefore overcoming the nonlinearity caused by the projection model. This allows the use of realistic 3D motion models and easy incorporation of self-motion measurements. All nonlinearities are concentrated in the observation model so that each particle projects a few tens of special points onto the image, on (and around) the 3D object’s surface. The likelihood of each state is then evaluated by comparing the color distributions inside and outside the object’s occluding contour. Since only pixel access operations are required, the method does not require the use of image processing routines like edge/feature extraction, color segmentation or 3D reconstruction, which can be sensitive to motion blur and optical distortions typical in applications of omnidirectional sensors to robotics. We show tracking applications considering different objects (balls, boxes), several projection models (catadioptric, dioptric, perspective) and several challenging scenarios (clutter, occlusion, illumination changes, motion and optical blur). We compare our methodology against a state-of-the-art alternative, both in realistic tracking sequences and with ground truth generated data.     

Contour-based object tracking with occlusion handling in video acquired using mobile cameras

We propose a tracking method which tracks the complete object regions, adapts to changing visual features, and handles occlusions. Tracking is achieved by evolving the contour from frame to frame by minimizing some energy functional evaluated in the contour vicinity defined by a band. Our approach has two major components related to the visual features and the object shape. Visual features (color, texture) are modeled by semiparametric models and are fused using independent opinion polling. Shape priors consist of shape level sets and are used to recover the missing object regions during occlusion. We demonstrate the performance of our method in real sequences with and without object occlusions.     

Fast occluded object tracking by a robust appearance filter

We propose a new method for object tracking in image sequences using template matching. To update the template, appearance features are smoothed temporally by robust Kalman filters, one to each pixel. The resistance of the resulting template to partial occlusions enables the accurate detection and handling of more severe occlusions. Abrupt changes of lighting conditions can also be handled, especially when photometric invariant color features are used, The method has only a few parameters and is computationally fast enough to track objects in real time.     

Using color reach histogram for object search in colour and/or depth scene

A method is proposed for object search in cluttered color and/or depth scene, which is based on color reach histogram. The histogram features an image by the distribution of pixel colors with their two- and three-dimensional locations. In real situation, it is sometimes difficult to keep illumination unchanged or stable, and then color distributions can address an another effective cue for object identification. The color reach histograms reveal strong robustness for illumination change and/or on-plane rotation. In comparison with conventional histogram-based methods, the stability and robustness in object search and tracking could be obtained and verified through many experiments with real objects.     

自适应嵌套级联的在线集成学习方法研究

针对视频目标检测问题,提出一种新的在线集成学习方法。该方法把目标检测看成两类分类问题,首先用少量已标注样本离线训练一个初始集成分类器,然后在检测目标的同时通过跟踪过滤虚警目标,并通过样本置信度作进一步验证自动标注样本,最后通过在线集成学习方法更新级联分类器。该方法通过在线调整级联分类器,提高分类器对目标环境变化的适应能力,在大量视频序列上进行实验验证,并与现有在线集成学习方法进行比较,结果表明,通过该方法训练得到的检测器不但能够很好地应对目标特征的变化,也能在出现目标遮挡及背景干扰下稳定地检测出目标,具有较好的适应性及鲁棒性。     

3D Object Tracking with Adaptively Weighted Local Bundles

The 3D object tracking from a monocular RGB image is a challenging task.Although popular color and edge-based methods have been well studied,they are only applicable to certain cases and new solutions to the challenges in real environment must be developed.In this paper,we propose a robust 3D object tracking method with adaptively weighted local bundles called AWLB tracker to handle more complicated cases.Each bundle represents a local region containing a set of local features.To alleviate the negative effect of the features in low-confidence regions,the bundles are adaptively weighted using a spatially-variant weighting function based on the confidence values of the involved energy terms.Therefore,in each frame,the weights of the energy items in each bundle are adapted to different situations and different regions of the same frame.Experiments show that the proposed method can improve the overall accuracy in challenging cases.We then verify the effectiveness of the proposed confidence-based adaptive weighting method using ablation studies and show that the proposed method overperforms the existing single-feature methods and multi-feature methods without adaptive weighting.     

字典学习联合粒子滤波鲁棒跟踪

针对运动目标鲁棒跟踪问题,提出一种基于离线字典学习的视频目标跟踪鲁棒算法。采用字典编码方式提取目标的局部区域描述符,随后通过训练分类器将跟踪问题转化为背景和前景分类问题,最终通过粒子滤波对物体位置进行估计实现跟踪。该算法能够有效解决由于光照变化、背景复杂、快速运动、遮挡产生的跟踪困难。经过不同图像序列的实验对比表明,与现有方法相比,本文算法的鲁棒性较高。     

Exploring legibility of augmented reality X-ray

Virtual objects can be visualized inside real objects using augmented reality (AR). This visualization is called AR X-ray because it gives the impression of seeing through the real object. In standard AR, virtual information is overlaid on top of the real world. To position a virtual object inside an object, AR X-ray requires partially occluding the virtual object with visually important regions of the real object. In effect, the virtual object becomes less legible compared to when it is completely unoccluded. Legibility is an important consideration for various applications of AR X-ray. In this research, we explored legibility in two implementations of AR X-ray, namely, edge-based and saliency-based. In our first experiment, we explored on the tolerable amounts of occlusion to comfortably distinguish small virtual objects. In our second experiment, we compared edge-based and saliency-based AR X-ray methods when visualizing virtual objects inside various real objects. Moreover, we benchmarked the legibility of these two methods against alpha blending. From our experiments, we observed that users have varied preferences for proper amounts of occlusion cues for both methods. The partial occlusions generated by the edge-based and saliency-based methods need to be adjusted depending on the lighting condition and the texture complexity of the occluding object. In most cases, users identify objects faster with saliency-based AR X-ray than with edge-based AR X-ray. Insights from this research can be directly applied to the development of AR X-ray applications.     

Contour based object detection using part bundles

In this paper we propose a novel framework for contour based object detection from cluttered environments. Given a contour model for a class of objects, it is first decomposed into fragments hierarchically. Then, we group these fragments into part bundles, where a part bundle can contain overlapping fragments. Given a new image with set of edge fragments we develop an efficient voting method using local shape similarity between part bundles and edge fragments that generates high quality candidate part configurations. We then use global shape similarity between the part configurations and the model contour to find optimal configuration. Furthermore, we show that appearance information can be used for improving detection for objects with distinctive texture when model contour does not sufficiently capture deformation of the objects.     

Small object detection in cluttered image using a correlation based active contour model

This paper presents a novel energy function for active contour models based on autocorrelation function, which is capable of detecting small objects against a cluttered background. In the proposed method, image features are calculated using a combination of short-term autocorrelations (STA) computed from the image pixels to represent region information. The obtained features are exploited to define an energy function for the localized region-based active contour model called normalized accumulated short-term autocorrelation (NASTA). Minimizing this energy function, we can accurately detect small objects in images containing cluttered and textured backgrounds. Moreover, the proposed method provides high robustness against random noise and can precisely locate small objects in noisy backgrounds, difficult to be detected with naked eye. Experimental results indicate remarkable advantages of our approach comparing to existing methods.     

自适应融合颜色和深度信息的人体轮廓跟踪

采用活动轮廓对人体目标建模,提出一 种新的水平集框架下自适应融合RGB-D图像的颜色和深度信息的人体轮廓跟踪方法. 设计了一种基于超像素的局部自适应权重计算方法,自动确定深度信息在水平集演化中的重要性. 基于深度信息的活动轮廓驱动外力包括由边缘生成的梯度向量流和由目标/背景深度模型生成的置信图,基于颜色信息的驱动外力由目标/背景颜色模型生成的置信图,这三种外力通过局部自适应权重融合,驱动活动轮廓向目标的边界演化.为了得到更加精确的目标轮廓和防止误差漂移,基于本文观察到的人体表面在深度图像中的两个特性,提出两个简单但有效的算法对水平集方法得到的结果进行精化调整. 最后,通过实验验证了本文算法的优越性.     

A color-based particle filter for multiple object tracking in an outdoor environment

Tracking multiple objects is more challenging than tracking a single object. Some problems arise in multiple-object tracking that do not exist in single-object tracking, such as object occlusion, the appearance of a new object and the disappearance of an existing object, updating the occluded object, etc. In this article, we present an approach to handling multiple-object tracking in the presence of occlusions, background clutter, and changing appearance. The occlusion is handled by considering the predicted trajectories of the objects based on a dynamic model and likelihood measures. We also propose target-model-update conditions, ensuring the proper tracking of multiple objects. The proposed method is implemented in a probabilistic framework such as a particle filter in conjunction with a color feature. The particle filter has proven very successful for nonlinear and non-Gaussian estimation problems. It approximates a posterior probability density of the state, such as the object’s position, by using samples or particles, where each state is denoted as the hypothetical state of the tracked object and its weight. The observation likelihood of the objects is modeled based on a color histogram. The sample weight is measured based on the Bhattacharya coefficient, which measures the similarity between each sample’s histogram and a specified target model. The algorithm can successfully track multiple objects in the presence of occlusion and noise. Experimental results show the effectiveness of our method in tracking multiple objects.     

融合颜色和增量形状先验的目标轮廓跟踪

基于主动轮廓的跟踪方法被广泛应用于移动摄像机下运动物体轮廓的跟踪. 针对传统方法容易受噪音、部分遮挡、背景干扰等因素影响的缺点, 提出了一个分层的基于水平集(Level sets)的跟踪框架. 该框架将颜色信息和形状先验有效地结合起来. 在框架的第一层, 初始轮廓首先根据颜色信息进化, 通过引入一个反映邻域像素之间关系的惩罚因子来改进传统的速度模型. 然后, 基于Mahalanobis距离的判别式被用来决定是否需要引入形状先验, 如果不需要, 则第一层基于颜色进化的结果就作为最终的跟踪结果; 否则, 第一层得到的轮廓需要在第二层中在形状先验的约束下继续进化. 在第二层轮廓进化中, 本文提出了一个权重形状距离因子(Weighted shape distance term, WSDT), 用来融合全局的形状信息和局部的颜色信息. 形状先验模型建立在主成分分析(Principal component analysis, PCA)子空间并通过增量学习算法在线更新. 实验结果证明了方法的有效性和鲁棒性.     

Semantic part segmentation method based 3D object pose estimation with RGB-D images for bin-picking

3D object pose estimation for grasping and manipulation is a crucial task in robotic and industrial applications. Robustness and efficiency for robotic manipulation are desirable properties that are still very challenging in complex and cluttered scenes, because 3D objects have different appearances, illumination and occlusion when seen from different viewpoints. This article proposes a Semantic Point Pair Feature (PPF) method for 3D object pose estimation, which combines the semantic image segmentation using deep learning with the voting-based 3D object pose estimation. The Part Mask RCNN ispresented to obtain the semantic object-part segmentation related to the point cloud of object, which is combined with the PPF method for 3D object pose estimation. In order to reduce the cost of collecting datasets in cluttered scenes, a physically-simulated environment is constructed to generate labeled synthetic semantic datasets. Finally, two robotic bin-picking experiments are demonstrated and the Part Mask RCNN for scene segmentation is evaluated through the constructed 3D object datasets. The experimental results show that the proposed Semantic PPF methodimproves the robustness and efficiency of 3D object pose estimation in cluttered scenes with partial occlusions.     

Weakly supervised detection with decoupled attention-based deep representation

Training object detectors with only image-level annotations is an important problem with a variety of applications. However, due to the deformable nature of objects, a target object delineated by a bounding box always includes irrelevant context and occlusions, which causes large intra-class object variations and ambiguity in object-background distinction. For this reason, identifying the object of interest from a substantial amount of cluttered backgrounds is very challenging. In this paper, we propose a decoupled attention-based deep model to optimize region-based object representation. Different from existing approaches posing object representation in a single-tower model, our proposed network decouples object representation into two separate modules, i.e., image representation and attention localization. The image representation module captures content-based semantic representation, while the attention localization module regresses an attention map which simultaneously highlights the locations of the discriminative object parts and down weights the irrelevant backgrounds presented in the image. The combined representation alleviates the impact from the noisy context and occlusions inside an object bounding box. As a result, object-background ambiguity can be largely reduced and background regions can be suppressed effectively. In addition, the proposed object representation model can be seamlessly integrated into a state-of-the-art weakly supervised detection framework, and the entire model can be trained end-to-end. We extensively evaluate the detection performance on the PASCAL VOC 2007, VOC 2010 and VOC2012 datasets. Experimental results demonstrate that our approach effectively improves weakly supervised object detection.     

融合SPA遮挡分割的多目标跟踪方法

复杂环境下的多目标视频跟踪是计算机视觉领域的一个难点,有效处理目标间遮挡是解决多目标跟踪问题的关键。将运动分割方法引入目标跟踪领域,提出一种融合骨架点指派(SPA)遮挡分割的多目标跟踪方法。由底层光流信息得到骨架点,并估计骨架点遮挡状态;综合使用目标外观、运动、颜色信息等高级语义信息,将骨架点指派给各个目标;最后以骨架点为核,对运动前景密集分类,得到准确的目标前景像素;在粒子滤波器跟踪框架下,使用概率外观模型进行多目标跟踪。在PETS2009数据集上的实验结果表明,文中方法能够改进现有多目标跟踪方法对目标间交互适应性较差的缺点,更好地处理动态遮挡问题。     

Fuzzy spatial constraints and ranked partitioned sampling approach for multiple object tracking

While particle filters are now widely used for object tracking in videos, the case of multiple object tracking still raises a number of issues. Among them, a first, and very important, problem concerns the exponential increase of the number of particles with the number of objects to be tracked, that can make some practical applications intractable. To achieve good tracking performances, we propose to use a Partitioned Sampling method in the estimation process with an additional feature about the ordering sequence in which the objects are processed. We call it Ranked Partitioned Sampling, where the optimal order in which objects should be processed and tracked is estimated jointly with the object state. Another essential point concerns the modeling of possible interactions between objects. As another contribution, we propose to represent these interactions within a formal framework relying on fuzzy sets theory. This allows us to easily model spatial constraints between objects, in a general and formal way. The association of these two contributions was tested on typical videos exhibiting difficult situations such as partial or total occlusions, and appearance or disappearance of objects. We show the benefit of using conjointly these two contributions, in comparison to classical approaches, through multiple object tracking and articulated object tracking experiments on real video sequences. The results show that our approach provides less tracking errors than those obtained with the classical Partitioned Sampling method, without the need for increasing the number of particles.     

Shape Based Detection and Top-Down Delineation Using Image Segments

We introduce a segmentation-based detection and top-down figure-ground delineation algorithm. Unlike common methods which use appearance for detection, our method relies primarily on the shape of objects as is reflected by their bottom-up segmentation. Our algorithm receives as input an image, along with its bottom-up hierarchical segmentation. The shape of each segment is then described both by its significant boundary sections and by regional, dense orientation information derived from the segment’s shape using the Poisson equation. Our method then examines multiple, overlapping segmentation hypotheses, using their shape and color, in an attempt to find a “coherent whole,” i.e., a collection of segments that consistently vote for an object at a single location in the image. Once an object is detected, we propose a novel pixel-level top-down figure-ground segmentation by “competitive coverage” process to accurately delineate the boundaries of the object. In this process, given a particular detection hypothesis, we let the voting segments compete for interpreting (covering) each of the semantic parts of an object. Incorporating competition in the process allows us to resolve ambiguities that arise when two different regions are matched to the same object part and to discard nearby false regions that participated in the voting process. We provide quantitative and qualitative experimental results on challenging datasets. These experiments demonstrate that our method can accurately detect and segment objects with complex shapes, obtaining results comparable to those of existing state of the art methods. Moreover, our method allows us to simultaneously detect multiple instances of class objects in images and to cope with challenging types of occlusions such as occlusions by a bar of varying size or by another object of the same class, that are difficult to handle with other existing class-specific top-down segmentation methods.