Detecting motion regions in presence of strong parallax from a moving camera by multi-view geometric constraints

We present a method for detecting motion regions in video sequences observed by a moving camera in the presence of a strong parallax due to static 3D structures. The proposed method classifies each image pixel into planar background, parallax, or motion regions by sequentially applying 2D planar homographies, the epipolar constraint, and a novel geometric constraint called the "structure consistency constraint." The structure consistency constraint, being the main contribution of this paper, is derived from the relative camera poses in three consecutive frames and is implemented within the "Plane + Parallax" framework. Unlike previous planar-parallax constraints proposed in the literature, the structure consistency constraint does not require the reference plane to be constant across multiple views. It directly measures the inconsistency between the projective structures from the same point under camera motion and reference plane change. The structure consistency constraint is capable of detecting moving objects followed by a moving camera in the same direction, a so-called degenerate configuration where the epipolar constraint fails. We demonstrate the effectiveness and robustness of our method with experimental results of real-world video sequences.     

Stereo geometry from 3D ego-motion streams

This paper addresses the problem of geometry determination of a stereo rig that undergoes general rigid motions. Neither known reference objects nor stereo correspondence are required. With almost no exception, all existing online solutions attempt to recover stereo geometry by first establishing stereo correspondences. We first describe a mathematical framework that allows us to solve for stereo geometry, i.e., the rotation and translation between the two cameras, using only motion correspondence that is far easier to acquire than stereo correspondence. Second, we show how to recover the rotation and present two linear methods, as well as a nonlinear one to solve for the translation. Third, we perform a stability study for the developed methods in the presence of image noise, camera parameter noise, and ego-motion noise. We also address accuracy issues. Experiments with real image data are presented. The work allows the concept of online calibration to be broadened, as it is no longer true that only single cameras can exploit structure-from-motion strategies; even the extrinsic parameters of a stereo rig of cameras can do so without solving stereo correspondence. The developed framework is applicable for estimating the relative three-dimensional (3D) geometry associated with a wide variety of mounted devices used in vision and robotics, by exploiting their scaled ego-motion streams.     

Direct recovery of planar-parallax from multiple frames

We present an algorithm that estimates dense planar-parallax motion from multiple uncalibrated views of a 3D scene. This generalizes the "plane+parallax" recovery methods to more than two frames. The parallax motion of pixels across multiple frames (relative to a planar surface) is related to the 3D scene structure and the camera epipoles. The parallax field, the epipoles, and the 3D scene structure are estimated directly from image brightness variations across multiple frames, without precomputing correspondences.     

Independent motion detection in 3D scenes

This paper presents an algorithmic approach to the problem of detecting independently moving objects in 3D scenes that are viewed under camera motion. There are two fundamental constraints that can be exploited for the problem: 1) two/multiview camera motion constraint (for instance, the epipolar/trilinear constraint) and 2) shape constancy constraint. Previous approaches to the problem either use only partial constraints, or rely on dense correspondences or flow. We employ both the fundamental constraints in an algorithm that does not demand a priori availability of correspondences or flow. Our approach uses the plane-plus-parallax decomposition to enforce the two constraints. It is also demonstrated that for a class of scenes, called sparse 3D scenes in which genuine parallax and independent motions may be confounded, how the plane-plus-parallax decomposition allows progressive introduction, and verification of the fundamental constraints. Results of the algorithm on some difficult sparse 3D scenes are promising.     

Observability of 3D Motion

This paper examines the inherent difficulties in observing 3D rigid motion from image sequences. It does so without considering a particular estimator. Instead, it presents a statistical analysis of all the possible computational models which can be used for estimating 3D motion from an image sequence. These computational models are classified according to the mathematical constraints that they employ and the characteristics of the imaging sensor (restricted field of view and full field of view). Regarding the mathematical constraints, there exist two principles relating a sequence of images taken by a moving camera. One is the epipolar constraint, applied to motion fields, and the other the positive depth constraint, applied to normal flow fields. 3D motion estimation amounts to optimizing these constraints over the image. A statistical modeling of these constraints leads to functions which are studied with regard to their topographic structure, specifically as regards the errors in the 3D motion parameters at the places representing the minima of the functions. For conventional video cameras possessing a restricted field of view, the analysis shows that for algorithms in both classes which estimate all motion parameters simultaneously, the obtained solution has an error such that the projections of the translational and rotational errors on the image plane are perpendicular to each other. Furthermore, the estimated projection of the translation on the image lies on a line through the origin and the projection of the real translation. The situation is different for a camera with a full (360 degree) field of view (achieved by a panoramic sensor or by a system of conventional cameras). In this case, at the locations of the minima of the above two functions, either the translational or the rotational error becomes zero, while in the case of a restricted field of view both errors are non-zero. Although some ambiguities still remain in the full field of view case, the implication is that visual navigation tasks, such as visual servoing, involving 3D motion estimation are easier to solve by employing panoramic vision. Also, the analysis makes it possible to compare properties of algorithms that first estimate the translation and on the basis of the translational result estimate the rotation, algorithms that do the opposite, and algorithms that estimate all motion parameters simultaneously, thus providing a sound framework for the observability of 3D motion. Finally, the introduced framework points to new avenues for studying the stability of image-based servoing schemes.     

Real-time motion detection based on SW/HW-codesign for walking rescue robots

In a rescue operation walking robots offer a great deal of flexibility in traversing uneven terrain in an uncontrolled environment. For such a rescue robot, each motion is a potential vital sign and the robot should be sensitive enough to detect such motion, at the same time maintaining high accuracy to avoid false alarms. However, the existing techniques for motion detection have severe limitations in dealing with strong levels of ego-motion on walking robots. This paper proposes an optical flow-based method for the detection of moving objects using a single camera mounted on a hexapod robot. The proposed algorithm estimates and compensates ego-motion to allow for object detection from a continuously moving robot, using a first-order-flow motion model. Our algorithm can deal with strong rotation and translation in 3D, with four degrees of freedom. Two alternative object detection methods using a 2D-histogram based vector clustering and motion-compensated frame differencing, respectively, are examined for the detection of slow- and fast-moving objects. The FPGA implementation with optimized resource utilization using SW/HW codesign can process video frames in real-time at 31 fps. The new algorithm offers a significant improvement in performance over the state-of-the-art, under harsh environment and performs equally well under smooth motion.     

由抖动图象序列建立自然环境三维真实感全景模型

由真实环境中的现场图象进行三维环境建模是目前国际上研究的热点问题。本文依据合理的运动模型，提出和实现了由包含抖动的摄像机运动下的图象序列建立３Ｄ环境全景模型的两步法。首先通过运动滤波和运动分解获得运动稳定的图象序列，然后采用无特征提取的时空纹理方向精确估计、深度边界确定和遮挡恢复算法，建立全局自然景物的真实感三维环境模型。提出了２种三维全景图象的表示方法，即非阵列方式深度分层区域表示和阵列方式的深度分层布景表示，可用于机器人全局定位的自然路标提取和真实环境虚拟再现的图象合成。该研究推广和结合了外极面图象的方法和全景图象的方法，放宽了对运动的要求，从而可使该种方法适用于室外颠簸的道路环境。和现有运动分层方法相比，避免了该类方法迭代过程中的局部最小化问题，并具有计算和存储效率高，适应性强，算法鲁棒性好的优点。     

Silhouette coherence for camera calibration under circular motion

We present a new approach to camera calibration as a part of a complete and practical system to recover digital copies of sculpture from uncalibrated image sequences taken under turntable motion. In this paper, we introduce the concept of the silhouette coherence of a set of silhouettes generated by a 3D object. We show how the maximization of the silhouette coherence can be exploited to recover the camera poses and focal length. Silhouette coherence can be considered as a generalization of the well-known epipolar tangency constraint for calculating motion from silhouettes or outlines alone. Further, silhouette coherence exploits all the geometric information encoded in the silhouette (not just at epipolar tangency points) and can be used in many practical situations where point correspondences or outer epipolar tangents are unavailable. We present an algorithm for exploiting silhouette coherence to efficiently and reliably estimate camera motion. We use this algorithm to reconstruct very high quality 3D models from uncalibrated circular motion sequences, even when epipolar tangency points are not available or the silhouettes are truncated. The algorithm has been integrated into a practical system and has been tested on more than 50 uncalibrated sequences to produce high quality photo-realistic models. Three illustrative examples are included in this paper. The algorithm is also evaluated quantitatively by comparing it to a state-of-the-art system that exploits only epipolar tangents     

Epipolar geometry from profiles under circular motion

Addresses the problem of motion estimation from profiles (apparent contours) of an object rotating on a turntable in front of a single camera. A practical and accurate technique for solving this problem from profiles alone is developed. It is precise enough to reconstruct the shape of the object. No correspondences between points or lines are necessary. Symmetry of the surface of revolution swept out by the rotating object is exploited to obtain the image of the rotation axis and the homography relating epipolar lines in two views robustly and elegantly. These, together with geometric constraints for images of rotating objects, are used to obtain first the image of the horizon, which is the projection of the plane that contains the camera centers, and then the epipoles, thus fully determining the epipolar geometry of the image sequence. The estimation of this geometry by this sequential approach avoids many of the problems found in other algorithms. The search for the epipoles, by far the most critical step, is carried out as a simple 1D optimization. Parameter initialization is trivial and completely automatic at all stages. After the estimation of the epipolar geometry, the Euclidean motion is recovered using the fixed intrinsic parameters of the camera obtained either from a calibration grid or from self-calibration techniques. Finally, the spinning object is reconstructed from its profiles using the motion estimated in the previous stage. Results from real data are presented, demonstrating the efficiency and usefulness of the proposed methods     

基于摄像机纵向运动的序列图像的实时漫游

对于摄像机纵向运动的序列图像，提出了一种基于极线几何约束关系的当前视点目标图像生成算法：1）利用基于傅立叶变换的方法，得到远视点源图像中和近视点源图像无对应点的区域，并用一种背景色将其填充，得到填充后的远视点源图像；2）利用极线的整体匹配性质在两幅源图像中确定遍历整个源图像的对应源极线；3）将所有对应源极线按灰度进行分段；4）用动态规划匹配法确定对应源极线上段与段之间的匹配关系；5）通过两条对应源极线插值合成一条目标极线，生成当前视点的目标图像；6）对上一步得到的目标图像的背景色区域加以处理，得到最终的当前视点目标图像．提出了一种链表数据结构存储每相邻两幅源图像之间的预处理信息．漫游时实时读取．开发了一个图像漫游器，使用户能在其中实现不同视点的实时漫游．     

A review and evaluation of methods estimating ego-motion

If a visual observer moves through an environment, the patterns of light that impinge its retina vary leading to changes in sensed brightness. Spatial shifts of brightness patterns in the 2D image over time are called optic flow. In contrast to optic flow visual motion fields denote the displacement of 3D scene points projected onto the camera’s sensor surface. For translational and rotational movement through a rigid scene parametric models of visual motion fields have been defined. Besides ego-motion these models provide access to relative depth, and both ego-motion and depth information is useful for visual navigation.In the past 30 years methods for ego-motion estimation based on models of visual motion fields have been developed. In this review we identify five core optimization constraints which are used by 13 methods together with different optimization techniques.¹ In the literature methods for ego-motion estimation typically have been evaluated by using an error measure which tests only a specific ego-motion. Furthermore, most simulation studies used only a Gaussian noise model. Unlike, we test multiple types and instances of ego-motion. One type is a fixating ego-motion, another type is a curve-linear ego-motion. Based on simulations we study properties like statistical bias, consistency, variability of depths, and the robustness of the methods with respect to a Gaussian or outlier noise model. In order to achieve an improvement of estimates for noisy visual motion fields, part of the 13 methods are combined with techniques for robust estimation like m-functions or RANSAC. Furthermore, a realistic scenario of a stereo image sequence has been generated and used to evaluate methods of ego-motion estimation provided by estimated optic flow and depth information.     

Simultaneous two-view epipolar geometry estimation and motion segmentation by 4D tensor voting

We address the problem of simultaneous two-view epipolar geometry estimation and motion segmentation from nonstatic scenes. Given a set of noisy image pairs containing matches of n objects, we propose an unconventional, efficient, and robust method, 4D tensor voting, for estimating the unknown n epipolar geometries, and segmenting the static and motion matching pairs into n, independent motions. By considering the 4D isotropic and orthogonal joint image space, only two tensor voting passes are needed, and a very high noise to signal ratio (up to five) can be tolerated. Epipolar geometries corresponding to multiple, rigid motions are extracted in succession. Only two uncalibrated frames are needed, and no simplifying assumption (such as affine camera model or homographic model between images) other than the pin-hole camera model is made. Our novel approach consists of propagating a local geometric smoothness constraint in the 4D joint image space, followed by global consistency enforcement for extracting the fundamental matrices corresponding to independent motions. We have performed extensive experiments to compare our method with some representative algorithms to show that better performance on nonstatic scenes are achieved. Results on challenging data sets are presented.     

Scalable Extrinsic Calibration of Omni-Directional Image Networks

We describe a linear-time algorithm that recovers absolute camera orientations and positions, along with uncertainty estimates, for networks of terrestrial image nodes spanning hundreds of meters in outdoor urban scenes. The algorithm produces pose estimates globally consistent to roughly 0.1° (2 milliradians) and 5 centimeters on average, or about four pixels of epipolar alignment.We assume that adjacent nodes observe overlapping portions of the scene, and that at least two distinct vanishing points are observed by each node. The algorithm decouples registration into pure rotation and translation stages. The rotation stage aligns nodes to commonly observed scene line directions; the translation stage assigns node positions consistent with locally estimated motion directions, then registers the resulting network to absolute (Earth) coordinates.The paper's principal contributions include: extension of classic registration methodsto large scale and dimensional extent; a consistent probabilistic framework for modeling projective uncertainty; and a new hybrid of Hough transform and expectation maximization algorithms.We assess the algorithm's performance on synthetic and real data, and draw several conclusions. First, by fusing thousands of observations the algorithm achieves accurate registration even in the face of significant lighting variations, low-level feature noise, and error in initial pose estimates. Second, the algorithm's robustness and accuracy increase with image field of view. Third, the algorithm surmounts the usual tradeoff between speed and accuracy; it is both faster and more accurate than manual bundle adjustment.     

Motion estimation in image sequences using the deformation ofapparent contours

The problem of determining the camera motion from apparent contours or silhouettes of a priori unknown curved 3D surfaces is considered. In a sequence of images, it is shown how to use the generalized epipolar constraint on apparent contours. One such constraint is obtained for each epipolar tangency point in each image pair. An accurate algorithm for computing the motion is presented based on a maximum likelihood estimate. It is shown how to generate initial estimates on the camera motion using only the tracked contours. It is also shown that in theory the motion can be calculated from the deformation of a single contour. The algorithm has been tested on several real image sequences, for both Euclidean and projective reconstruction. The resulting motion estimate is compared to motion estimates calculated independently using standard feature-based methods. The motion estimate is also used to classify the silhouettes as curves or apparent contours. The statistical evaluation shows that the technique gives accurate and stable results     

Using vanishing points for camera calibration

In this article a new method for the calibration of a vision system which consists of two (or more) cameras is presented. The proposed method, which uses simple properties of vanishing points, is divided into two steps. In the first step, the intrinsic parameters of each camera, that is, the focal length and the location of the intersection between the optical axis and the image plane, are recovered from a single image of a cube. In the second step, the extrinsic parameters of a pair of cameras, that is, the rotation matrix and the translation vector which describe the rigid motion between the coordinate systems fixed in the two cameras are estimated from an image stereo pair of a suitable planar pattern. Firstly, by matching the corresponding vanishing points in the two images the rotation matrix can be computed, then the translation vector is estimated by means of a simple triangulation. The robustness of the method against noise is discussed, and the conditions for optimal estimation of the rotation matrix are derived. Extensive experimentation shows that the precision that can be achieved with the proposed method is sufficient to efficiently perform machine vision tasks that require camera calibration, like depth from stereo and motion from image sequence.     

3D Surface Reconstruction Using Occluding Contours

This paper addresses the problem of 3D surface reconstruction using image sequences. It has been shown that shape recovery from three or more occluding contours of the surface is possible given a known camera motion. Several algorithms, which have been recently proposed, allow such a reconstruction under the assumption of a linear camera motion. A new approach is presented which deals with the reconstruction problem directly from a discrete point of view. First, a theoretical study of the epipolar correspondence between occluding contours is achieved. A correct depth formulation is then derived from a local approximation of the surface up to order two. This allows the local shape to be estimated, given three consecutive contours, without any constraints on the camera motion. Experimental results are presented for both synthetic and real data.     

Threading fundamental matrices

We present a new function that operates on fundamental matrices across a sequence of views. The operation, we call “threading”, connects two consecutive fundamental matrices using the trifocal tensor as the connecting thread. The threading operation guarantees that consecutive camera matrices are consistent with a unique 3D model, without ever recovering a 3D model. Applications include recovery of camera ego-motion from a sequence of views, image stabilization across a sequence, and multi-view image based rendering     

A Variational Framework for Structure from Motion in Omnidirectional Image Sequences

We address the problem of depth and ego-motion estimation from omnidirectional images. We propose a correspondence-free structure-from-motion problem for sequences of images mapped on the 2-sphere. A novel graph-based variational framework is first proposed for depth estimation between pairs of images. The estimation is cast as a TV-L1 optimization problem that is solved by a fast graph-based algorithm. The ego-motion is then estimated directly from the depth information without explicit computation of the optical flow. Both problems are finally addressed together in an iterative algorithm that alternates between depth and ego-motion estimation for fast computation of 3D information from motion in image sequences. Experimental results demonstrate the effective performance of the proposed algorithm for 3D reconstruction from synthetic and natural omnidirectional images.     

Moving object detection and segmentation in urban environments from a moving platform

This paper proposes an effective approach to detect and segment moving objects from two time-consecutive stereo frames, which leverages the uncertainties in camera motion estimation and in disparity computation. First, the relative camera motion and its uncertainty are computed by tracking and matching sparse features in four images. Then, the motion likelihood at each pixel is estimated by taking into account the ego-motion uncertainty and disparity in computation procedure. Finally, the motion likelihood, color and depth cues are combined in the graph-cut framework for moving object segmentation. The efficiency of the proposed method is evaluated on the KITTI benchmarking datasets, and our experiments show that the proposed approach is robust against both global (camera motion) and local (optical flow) noise. Moreover, the approach is dense as it applies to all pixels in an image, and even partially occluded moving objects can be detected successfully. Without dedicated tracking strategy, our approach achieves high recall and comparable precision on the KITTI benchmarking sequences.