Rigid body segmentation and shape description from dense opticalflow under weak perspective

We present an algorithm for identifying and tracking independently moving rigid objects from optical flow. Some previous attempts at segmentation via optical flow have focused on finding discontinuities in the flow field. While discontinuities do indicate a change in scene depth, they do not in general signal a boundary between two separate objects. The proposed method uses the fact that each independently moving object has a unique epipolar constraint associated with its motion. Thus motion discontinuities based on self-occlusion can be distinguished from those due to separate objects. The use of epipolar geometry allows for the determination of individual motion parameters for each object as well as the recovery of relative depth for each point on the object. The algorithm assumes an affine camera where perspective effects are limited to changes in overall scale. No camera calibration parameters are required. A Kalman filter based approach is used for tracking motion parameters with time     

On 3-D scene flow and structure recovery from multiview image sequences

Two novel systems computing dense three-dimensional (3-D) scene flow and structure from multiview image sequences are described in this paper. We do not assume rigidity of the scene motion, thus allowing for nonrigid motion in the scene. The first system, integrated model-based system (IMS), assumes that each small local image region is undergoing 3-D affine motion. Non-linear motion model fitting based on both optical flow constraints and stereo constraints is then carried out on each local region in order to simultaneously estimate 3-D motion correspondences and structure. The second system is based on extended gradient-based system (EGS), a natural extension of two-dimensional (2-D) optical flow computation. In this method, a new hierarchical rule-based stereo matching algorithm is first developed to estimate the initial disparity map. Different available constraints under a multiview camera setup are further investigated and utilized in the proposed motion estimation. We use image segmentation information to adopt and maintain the motion and depth discontinuities. Within the framework for EGS, we present two different formulations for 3-D scene flow and structure computation. One formulation assumes that initial disparity map is accurate, while the other does not. Experimental results on both synthetic and real imagery demonstrate the effectiveness of our 3-D motion and structure recovery schemes. Empirical comparison between IMS and EGS is also reported.     

Estimating the focus of expansion in a video sequence using the trajectories of interest points

In this paper, we present a new algorithm for the computation of the focus of expansion in a video sequence. Although several algorithms have been proposed in the literature for its computation, almost all of them are based on the optical flow vectors between a pair of consecutive frames, so being very sensitive to noise, optical flow errors and camera vibrations. Our algorithm is based on the computation of the vanishing point of point trajectories, thus integrating information for more than two consecutive frames. It can improve performance in the presence of erroneous correspondences and occlusions in the field of view of the camera.The algorithm has been tested with virtual sequences generated with Blender, as well as some real sequences from both, the public KITTI benchmark, and a number of challenging video sequences also proposed in this paper. For comparison purposes, some algorithms from the literature have also been implemented. The results show that the algorithm has proven to be very robust, outperforming the compared algorithms, specially in outdoor scenes, where the lack of texture can make optical flow algorithms yield inaccurate results. Timing evaluation proves that the proposed algorithm can reach up to 15fps, showing its suitability for real-time applications.     

A Stochastic Approach for Blurred Image Restoration and Optical Flow Computation on Field Image Sequence

The blur in target images caused by camera vibration due to robot motion or hand shaking and by object(s) moving in the background scene is different to deal with in the computer vision system.In this paper,the authors study the relation model between motion and blur in the case of object motion existing in video image sequence,and work on a practical computation algorithm for both motion analysis and blut image restoration.Combining the general optical flow and stochastic process,the paper presents and approach by which the motion velocity can be calculated from blurred images.On the other hand,the blurred image can also be restored using the obtained motion information.For solving a problem with small motion limitation on the general optical flow computation,a multiresolution optical flow algoritm based on MAP estimation is proposed. For restoring the blurred image ,an iteration algorithm and the obtained motion velocity are used.The experiment shows that the proposed approach for both motion velocity computation and blurred image restoration works well.     

A Database and Evaluation Methodology for Optical Flow

The quantitative evaluation of optical flow algorithms by Barron et al. (1994) led to significant advances in performance. The challenges for optical flow algorithms today go beyond the datasets and evaluation methods proposed in that paper. Instead, they center on problems associated with complex natural scenes, including nonrigid motion, real sensor noise, and motion discontinuities. We propose a new set of benchmarks and evaluation methods for the next generation of optical flow algorithms. To that end, we contribute four types of data to test different aspects of optical flow algorithms: (1) sequences with nonrigid motion where the ground-truth flow is determined by tracking hidden fluorescent texture, (2) realistic synthetic sequences, (3) high frame-rate video used to study interpolation error, and (4) modified stereo sequences of static scenes. In addition to the average angular error used by Barron et al., we compute the absolute flow endpoint error, measures for frame interpolation error, improved statistics, and results at motion discontinuities and in textureless regions. In October 2007, we published the performance of several well-known methods on a preliminary version of our data to establish the current state of the art. We also made the data freely available on the web at . Subsequently a number of researchers have uploaded their results to our website and published papers using the data. A significant improvement in performance has already been achieved. In this paper we analyze the results obtained to date and draw a large number of conclusions from them.     

Motion estimation and segmentation

In the general structure-from-motion (SFM) problem involving several moving objects in a scene, the essential first step is to segment moving objects independently. We attempt to deal with the problem of optical flow estimation and motion segmentation over a pair of images. We apply a mean field technique to determine optical flow and motion boundaries and present a deterministic algorithm. Since motion discontinuities represented by line process are embedded in the estimation of the optical flow, our algorithm provides accurate estimates of optical flow especially along motion boundaries and handles occlusion and multiple motions. We show that the proposed algorithm outperforms other well-known algorithms in terms of estimation accuracy and timing.     

Accurate optical flow computation under non-uniform brightness variations

In this paper, we present a very accurate algorithm for computing optical flow with non-uniform brightness variations. The proposed algorithm is based on a generalized dynamic image model (GDIM) in conjunction with a regularization framework to cope with the problem of non-uniform brightness variations. To alleviate flow constraint errors due to image aliasing and noise, we employ a reweighted least-squares method to suppress unreliable flow constraints, thus leading to robust estimation of optical flow. In addition, a dynamic smoothness adjustment scheme is proposed to efficiently suppress the smoothness constraint in the vicinity of the motion and brightness variation discontinuities, thereby preserving motion boundaries. We also employ a constraint refinement scheme, which aims at reducing the approximation errors in the first-order differential flow equation, to refine the optical flow estimation especially for large image motions. To efficiently minimize the resulting energy function for optical flow computation, we utilize an incomplete Cholesky preconditioned conjugate gradient algorithm to solve the large linear system. Experimental results on some synthetic and real image sequences show that the proposed algorithm compares favorably to most existing techniques reported in literature in terms of accuracy in optical flow computation with 100% density.     

The computation of cloud base height from paired whole-sky imaging cameras

We have developed a novel approach to the extraction of cloud base height (CBH) from pairs of whole-sky imagers (WSIs). The core problem is to spatially register cloud fields from widely separated WSIs; this complete, triangulation provides the CBH measurements. The wide camera separation and the self-similarity of clouds defeats standard matching algorithms when applied to static views of the sky. In response, we use optical flow methods that exploit the fact that modern WSIs provide image sequences. We will describe the algorithm, a confidence metric for its performance, a method to correct the severe projective effects of the WSI camera, and results on real data.     

On the Spatial Statistics of Optical Flow

We present an analysis of the spatial and temporal statistics of “natural” optical flow fields and a novel flow algorithm that exploits their spatial statistics. Training flow fields are constructed using range images of natural scenes and 3D camera motions recovered from hand-held and car-mounted video sequences. A detailed analysis of optical flow statistics in natural scenes is presented and machine learning methods are developed to learn a Markov random field model of optical flow. The prior probability of a flow field is formulated as a Field-of-Experts model that captures the spatial statistics in overlapping patches and is trained using contrastive divergence. This new optical flow prior is compared with previous robust priors and is incorporated into a recent, accurate algorithm for dense optical flow computation. Experiments with natural and synthetic sequences illustrate how the learned optical flow prior quantitatively improves flow accuracy and how it captures the rich spatial structure found in natural scene motion.     

Improving monocular visual SLAM in dynamic environments: an optical-flow-based approach

Visual Simultaneous Localization and Mapping (visual SLAM) has attracted more and more researchers in recent decades and many state-of-the-art algorithms have been proposed with rather satisfactory performance in static scenarios. However, in dynamic scenarios, the performance of current visual SLAM algorithms degrades significantly due to the disturbance of the dynamic objects. To address this problem, we propose a novel method which uses optical flow to distinguish and eliminate the dynamic feature points from the extracted ones using RGB images as the only input. The static feature points are fed into the visual SLAM system for the camera pose estimation. We integrate our method with the original ORB-SLAM system and validate the proposed method with the challenging dynamic sequences from the TUM dataset and our recorded office dataset. The whole system can work in real time. Qualitative and quantitative evaluations demonstrate that our method significantly improves the performance of ORB-SLAM in dynamic scenarios.     

一种基于线性亮度变化模型的鲁棒的光流算法

提出了一种鲁棒光流算法,用于计算光照强度、帧间运动速度及运动速度变化较大情况下的光流场。在梯度约束方程中嵌入了线性亮度变化模型,以提高大的光照强度变化下算法稳健性;将各向异性扩散方程引入空间方向平滑约束,以改善运动不连续处的流速计算精度,并依此建立了多尺度空间微分光流算法。参数的均衡化得到了线性尺度变化下的恒定能量函数。迭代运算引入运动补偿的概念,使亮度误差减小。实验结果表明,在光照强度和运动速度及速度变化较大时,本文算法具有很好的计算精度,并产生密度100%的光流场。     

Non Uniform Multiresolution Method for Optical Flow and Phase Portrait Models: Environmental Applications

In this paper we define a complete framework for processing large image sequences for a global monitoring of short range oceanographic and atmospheric processes. This framework is based on the use of a non quadratic regularization technique for optical flow computation that preserves flow discontinuities. We also show that using an appropriate tessellation of the image according to an estimate of the motion field can improve optical flow accuracy and yields more reliable flows. This method defines a non uniform multiresolution approach for coarse to fine grid generation. It allows to locally increase the resolution of the grid according to the studied problem. Each added node refines the grid in a region of interest and increases the numerical accuracy of the solution in this region. We make use of such a method for solving the optical flow equation with a non quadratic regularization scheme allowing the computation of optical flow field while preserving its discontinuities. The second part of the paper deals with the interpretation of the obtained displacement field. For this purpose a phase portrait model used along with a new formulation of the approximation of an oriented flow field allowing to consider arbitrary polynomial phase portrait models for characterizing salient flow features. This new framework is used for processing oceanographic and atmospheric image sequences and presents an alternative to complex physical modeling techniques.     

铁路视频序列的FOE的估计

在结构化场景的轨道交通中,车载视频观测因相机平移运动而呈现出图像内容以某点为中心向四周扩散的现象,该点被称为FOE (Focus of Expansion)。 当前计算FOE的算法对噪声敏感且计算量大,不能准确地计算铁路场景中的FOE。鉴于此,文中提出一种铁路视频序列的FOE估计方法。该方法首先利用金字塔光流法对检测的Harris角点进行跟踪和粗匹配,并在此基础上利用RANSAC算法进行精确的匹配,求得基础矩阵,然后提取图像中的极线束并计算FOE。实验结果表明,所提算法比Hough 直线求得的FOE误差小,适于实时应用。     

On the Fourier Properties of Discontinuous Motion

Retinal image motion and optical flow as its approximation are fundamental concepts in the field of vision, perceptual and computational. However, the computation of optical flow remains a challenging problem as image motion includes discontinuities and multiple values mostly due to scene geometry, surface translucency and various photometric effects such as reflectance. In this contribution, we analyze image motion in the frequency space with respect to motion discontinuities and translucence. We derive the frequency structure of motion discontinuities due to occlusion and we demonstrate its various geometrical properties. The aperture problem is investigated and we show that the information content of an occlusion almost always disambiguates the velocity of an occluding signal suffering from the aperture problem. In addition, the theoretical framework can describe the exact frequency structure of Non-Fourier motion and bridges the gap between Non-Fourier visual phenomena and their understanding in the frequency domain.     

A Variational Approach to Video Registration with Subspace Constraints

This paper addresses the problem of non-rigid video registration, or the computation of optical flow from a reference frame to each of the subsequent images in a sequence, when the camera views deformable objects. We exploit the high correlation between 2D trajectories of different points on the same non-rigid surface by assuming that the displacement of any point throughout the sequence can be expressed in a compact way as a linear combination of a low-rank motion basis. This subspace constraint effectively acts as a trajectory regularization term leading to temporally consistent optical flow. We formulate it as a robust soft constraint within a variational framework by penalizing flow fields that lie outside the low-rank manifold. The resulting energy functional can be decoupled into the optimization of the brightness constancy and spatial regularization terms, leading to an efficient optimization scheme. Additionally, we propose a novel optimization scheme for the case of vector valued images, based on the dualization of the data term. This allows us to extend our approach to deal with colour images which results in significant improvements on the registration results. Finally, we provide a new benchmark dataset, based on motion capture data of a flag waving in the wind, with dense ground truth optical flow for evaluation of multi-frame optical flow algorithms for non-rigid surfaces. Our experiments show that our proposed approach outperforms state of the art optical flow and dense non-rigid registration algorithms.     

室内动态环境下基于网格分割与双地图耦合的RGB-D SLAM算法

为解决室内动态环境下现有RGB-D SLAM（同步定位与地图创建）系统定位精度低、建图效果差的问题,提出一种基于网格分割与双地图耦合的RGB-D SLAM算法。基于单应运动补偿与双向补偿光流法,根据几何连通性与深度图像聚类结果实现网格化运动分割,同时保证算法的快速性。利用静态区域内的特征点最小化重投影误差对相机进行位置估计。结合相机位姿、RGB-D图像、网格化运动分割图像,同时构建场景的稀疏点云地图和静态八叉树地图并进行耦合,在关键帧上使用基于网格分割和八叉树地图光线遍历的方法筛选静态地图点,更新稀疏点云地图,保障定位精度。公开数据集和实际动态场景中的实验结果都表明,本文算法能够有效提升室内动态场景中的相机位姿估计精度,实现场景静态八叉树地图的实时构建和更新。此外,本文算法能够实时运行在标准CPU硬件平台上,无需GPU等额外计算资源。     

Multithreaded Hybrid Feature Tracking for Markerless Augmented Reality

We describe a novel markerless camera tracking approach and user interaction methodology for augmented reality (AR) on unprepared tabletop environments. We propose a real-time system architecture that combines two types of feature tracking. Distinctive image features of the scene are detected and tracked frame-to-frame by computing optical flow. In order to achieve real-time performance, multiple operations are processed in a synchronized multi-threaded manner: capturing a video frame, tracking features using optical flow, detecting distinctive invariant features, and rendering an output frame. We also introduce user interaction methodology for establishing a global coordinate system and for placing virtual objects in the AR environment by tracking a user's outstretched hand and estimating a camera pose relative to it. We evaluate the speed and accuracy of our hybrid feature tracking approach, and demonstrate a proof-of-concept application for enabling AR in unprepared tabletop environments, using bare hands for interaction.     

Optical flow estimation for motion-compensated compression

The computation of optical flow within an image sequence is one of the most widely used techniques in computer vision. In this paper, we present a new approach to estimate the velocity field for motion-compensated compression. It is derived by a nonlinear system using the direct temporal integral of the brightness conservation constraint equation or the Displaced Frame Difference (DFD) equation. To solve the nonlinear system of equations, an adaptive framework is used, which employs velocity field modeling, a nonlinear least-squares model, Gauss–Newton and Levenberg–Marquardt techniques, and an algorithm of the progressive relaxation of the over-constraint. The three criteria by which successful motion-compensated compression is judged are 1.) The fidelity with which the estimated optical flow matches the ground truth motion, 2.) The relative absence of artifacts and “dirty window” effects for frame interpolation, and 3.) The cost to code the motion vector field. We base our estimated flow field on a single minimized target function, which leads to motion-compensated predictions without incurring penalties in any of these three criteria. In particular, we compare our proposed algorithm results with those from Block-Matching Algorithms (BMA), and show that with nearly the same number of displacement vectors per fixed block size, the performance of our algorithm exceeds that of BMA in all the three above points. We also test the algorithm on synthetic and natural image sequences, and use it to demonstrate applications for motion-compensated compression.     

结合图像识别的移动增强现实系统设计与应用

目的 在移动互联网时代下,移动增强现实应用得到越来越快的发展。然而户外场景中存在许多相似结构的建筑,且手机的存储和计算能力有限,因此应用多集中于室内小范围环境,对于室外大规模复杂场景的适应性较弱。对此,建立一套基于云端图像识别的移动增强现实系统。方法 为解决相似特征的误匹配问题,算法中将重力信息加入到SURF和BRISK特征描述中去,构建Gravity-SURF和Gravity-BRISK特征描述。云端系统对增强信息进行有效管理,采用基于Gravity-SURF特征的VLAD方法对大规模图像进行识别;在智能终端上的应用中呈现识别图像的增强信息,并利用识别图像的Gravity-BRISK特征和光流结合的方法对相机进行跟踪,采用Unity3D渲染引擎实时绘制3维模型。结果 在包含重力信息的4 000幅户外图像的数据库中进行实验。采用结合重力信息的特征描述算法,能够增强具有相似特征的描述符的区分性,并提高匹配正确率。图像识别算法的识别率能达到88%以上,识别时间在420 ms左右;光流跟踪的RMS误差小于1.2像素,帧率能达到23 帧/s。结论 本文针对室外大规模复杂场景建立的基于图像识别的移动增强现实系统,能方便对不同应用的增强现实数据进行管理。系统被应用到谷歌眼镜和新闻领域上,不局限于单一的应用领域。结果表明,识别算法和跟踪注册算法能够满足系统的精度和实时性要求。     

动态场景下基于语义和光流约束的视觉同步定位与地图构建

针对动态场景下的定位与静态语义地图构建问题,提出了一种基于语义和光流约束的动态环境下的同步定位与地图构建（SLAM）算法,以降低动态物体对定位与建图的影响。首先,对于输入的每一帧,通过语义分割获得图像中物体的掩模,再通过几何方法过滤不符合极线约束的特征点;接着,结合物体掩模与光流计算出每个物体的动态概率,根据动态概率过滤特征点以得到静态特征点,再利用静态特征点进行后续的相机位姿估计;然后,基于RGB-D图片和物体动态概率建立静态点云,并结合语义分割建立语义八叉树地图。最后,基于静态点云与语义分割创建稀疏语义地图。公共TUM数据集上的测试结果表明,高动态场景下,所提算法与ORB-SLAM2相比,在绝对轨迹误差和相对位姿误差上能取得95%以上的性能提升,与DS-SLAM、DynaSLAM相比分别减小了41%和11%的绝对轨迹误差,验证了该算法在高动态场景中具有较好的定位精度和鲁棒性。地图构建的实验结果表明,所提算法创建了静态语义地图,与点云地图相比,稀疏语义地图的存储空间需求量降低了99%。