Optimal combination of multiple sensors including stereo vision

The statistical combination of information from multiple sources is considered. The particular needs of the target application, stereo vision, require that the formulation be adequate to deal with highly correlated errors and constraints, and that it deal naturally with geometrical data.     

Tightly-coupled stereo vision-aided inertial navigation using feature-based motion sensors

A tightly-coupled stereo vision-aided inertial navigation system is proposed in this work, as a synergistic incorporation of vision with other sensors. In order to avoid loss of information possibly resulting by visual preprocessing, a set of feature-based motion sensors and an inertial measurement unit are directly fused together to estimate the vehicle state. Two alternative feature-based observation models are considered within the proposed fusion architecture. The first model uses the trifocal tensor to propagate feature points by homography, so as to express geometric constraints among three consecutive scenes. The second one is derived by using a rigid body motion model applied to three-dimensional (3D) reconstructed feature points. A kinematic model accounts for the vehicle motion, and a Sigma-Point Kalman filter is used to achieve a robust state estimation in the presence of non-linearities. The proposed formulation is derived for a general platform-independent 3D problem, and it is tested and demonstrated with a real dynamic indoor data-set alongside of a simulation experiment. Results show improved estimates than in the case of a classical visual odometry approach and of a loosely-coupled stereo vision-aided inertial navigation system, even in GPS (Global Positioning System)-denied conditions and when magnetometer measurements are not reliable.     

Generating near-spherical range panoramas by fusing optical flow and stereo from a single-camera folded catadioptric rig

We design a novel “folded” spherical catadioptric rig (formed by two coaxially-aligned spherical mirrors of distinct radii and a single perspective camera) to recover near-spherical range panoramas (about 360° × 153°) from the fusion of depth given by optical flow and stereoscopy. We observe that for rigid motion that is parallel to a plane, optical flow and stereo generate nearly complementary distributions of depth resolution. While optical flow provides strong depth cues in the periphery and near the poles of the view-sphere, stereo generates reliable depth in a narrow band about the equator instead. We exploit this dual-modality principle by modeling (separately) the depth resolution of optical flow and stereo in order to fuse them later on a probabilistic spherical panorama. We achieve a desired vertical field-of-view and optical resolution by deriving a linearized model of the rig in terms of three parameters (radii of the two mirrors plus axial distance between the mirrors’ centers). We analyze the error due to the violation of the single viewpoint constraint and formulate additional constraints on the design to minimize this error. We evaluate our proposed method via a synthetic model and with real-world prototypes by computing dense spherical panoramas of depth from cluttered indoor environments after fusing the two modalities (stereo and optical flow).     

Rectified Surface Mosaics

We approach mosaicing as a camera tracking problem within a known parameterized surface. From a video of a camera moving within a surface, we compute a mosaic representing the texture of that surface, flattened onto a planar image. Our approach works by defining a warp between images as a function of surface geometry and camera pose. Globally optimizing this warp to maximize alignment across all frames determines the camera trajectory, and the corresponding flattened mosaic image. In contrast to previous mosaicing methods which assume planar or distant scenes, or controlled camera motion, our approach enables mosaicing in cases where the camera moves unpredictably through proximal surfaces, such as in medical endoscopy applications.     

Visual contact with catadioptric cameras

Time to contact or time to collision (TTC) is utmost important information for animals as well as for mobile robots because it enables them to avoid obstacles; it is a convenient way to analyze the surrounding environment. The problem of TTC estimation is largely discussed in perspective images. Although a lot of works have shown the interest of omnidirectional camera for robotic applications such as localization, motion, monitoring, few works use omnidirectional images to compute the TTC. In this paper, we show that TTC can be also estimated on catadioptric images. We present two approaches for TTC estimation using directly or indirectly the optical flow based on de-rotation strategy. The first, called “gradient based TTC”, is simple, fast and it does not need an explicit estimation of the optical flow. Nevertheless, this method cannot provide a TTC on each pixel, valid only for para-catadioptric sensors and requires an initial segmentation of the obstacle. The second method, called “TTC map estimation based on optical flow”, estimates TTC on each point on the image and provides the depth map of the environment for any obstacle in any direction and is valid for all central catadioptric sensors. Some results and comparisons in synthetic and real images will be given.     

Single-view-point omnidirectional catadioptric cone mirror imager

We present here a comprehensive imaging theory about cone mirrors in a single-view-point (SVP) configuration and show that an SVP cone mirror catadioptric system is not only practical but also has unique advantages for certain applications. We show its merits and weaknesses and how to build a workable system.     

基于差分修正的SGDM算法

当前, 应用广泛的一阶深度学习优化器包括学习率非自适应优化器和学习率自适应优化器, 前者以SGDM为代表, 后者以Adam为代表, 这两类方法都使用指数滑动平均法来估计总体的梯度. 然而使用指数滑动平均法来估计总体梯度是有偏差且具有滞后性的, 本文提出基于差分修正的SGDM算法——RSGDM算法. 我们的贡献主要有3点...     

Central catadioptric image processing with geodesic metric

Because of the distortions produced by the insertion of a mirror, catadioptric images cannot be processed similarly to classical perspective images. Now, although the equivalence between such images and spherical images is well known, the use of spherical harmonic analysis often leads to image processing methods which are more difficult to implement. In this paper, we propose to define catadioptric image processing from the geodesic metric on the unitary sphere. We show that this definition allows to adapt very simply classical image processing methods. We focus more particularly on image gradient estimation, interest point detection, and matching. More generally, the proposed approach extends traditional image processing techniques based on Euclidean metric to central catadioptric images. We show in this paper the efficiency of the approach through different experimental results and quantitative evaluations.     

一种DSP折反射全景图像实时展开方法

为了提高折反射全景图像展开速度,可采用查表来取代耗时的计算,但查找表需要大量的存储空间,于是采用分块预取方法。通过分块,将待处理目标图像块对应的查找表和原始图像分块预先存入DSP片内存储器并完成展开,可以降低存取访问开销,加快查表展开。实验结果表明,分块预取方法在对1024×768分辨率的原始全景图像展开为1152×256分辨率的目标全景图像时,展开数据可达每秒97帧,与不采用分块预取策略的查表展开方法相比,此方法要快近20倍。     

Markov random fields for catadioptric image processing

Images obtained with catadioptric sensors contain significant deformations which prevent the direct use of classical image treatments. Thus, Markov random fields (MRF) whose usefulness is now obvious for projective image processing, cannot be used directly on catadioptric images because of the inadequacy of the neighborhood. In this paper, we propose to define a new neighborhood for MRF by using the equivalence theorem developed for central catadioptric sensors. We show the importance of this adaptation for segmentation, image restoration and motion detection.     

Self-calibration of hybrid central catadioptric and perspective cameras

Hybrid central catadioptric and perspective cameras are desired in practice, because the hybrid camera system can capture large field of view as well as high-resolution images. However, the calibration of the system is challenging due to heavy distortions in catadioptric cameras. In addition, previous calibration methods are only suitable for the camera system consisting of perspective cameras and catadioptric cameras with only parabolic mirrors, in which priors about the intrinsic parameters of perspective cameras are required. In this work, we provide a new approach to handle the problems. We show that if the hybrid camera system consists of at least two central catadioptric and one perspective cameras, both the intrinsic and extrinsic parameters of the system can be calibrated linearly without priors about intrinsic parameters of the perspective cameras, and the supported central catadioptric cameras of our method can be more generic. In this work, an approximated polynomial model is derived and used for rectification of catadioptric image. Firstly, with the epipolar geometry between the perspective and rectified catadioptric images, the distortion parameters of the polynomial model can be estimated linearly. Then a new method is proposed to estimate the intrinsic parameters of a central catadioptric camera with the parameters in the polynomial model, and hence the catadioptric cameras can be calibrated. Finally, a linear self-calibration method for the hybrid system is given with the calibrated catadioptric cameras. The main advantage of our method is that it cannot only calibrate both the intrinsic and extrinsic parameters of the hybrid camera system, but also simplify a traditional nonlinear self-calibration of perspective cameras to a linear process. Experiments show that our proposed method is robust and reliable.     

中心折反射相机标定方法综述

近十几年来,计算机视觉越来越受研究者们的欢迎,特别是全景相机由于其具有较大的视场而被广泛应用到许多领域,包括视频监控、机器人导航、电视电话会议、场景重建以及虚拟现实等。摄像机标定是从二维图像获得三维信息必不可少的一步,摄像机标定结果的好坏直接决定着三维重建结果以及其它计算机视觉应用效果的好坏,所以,研究摄像机的标定方法具有重要的理论研究意义和重要的实际应用价值。这里将2000年到2012年折反射相机标定方法按照标定像的不同分为五大类：基于线的标定、基于二维标定块标定、基于三维点的标定、基于球的标定和自标定,且简要分析其优缺点。     

彩色图像对自适应匹配算法

提出了两种方法来提高图像匹配的精度。一种方法是改变自适应匹配窗口的选择方法,与原来的矩形窗口相比较,提出的窗口选取方法能在低纹理区域得到包含有效信息量更多、更接近实际边界的窗口。此外,通过降低边界点相似度在计算过程中的权重来降低边界点影响力,使得视差不连续区域的匹配精度得到提高。算法采用了Middlebury网站上提供的四幅立体图像对Tsukuba、Venus、Teddy和Cones分别进行实验验证。实验表明文中提出的两种方法对四幅图像的匹配精度均有所提高。     

Omni-directional stereo

Omnidirectional views of an indoor environment at different locations are integrated into a global map. A single camera swiveling about the vertical axis takes consecutive images and arranges them into a panoramic representation, which provides rich information around the observation point: a precise omnidirectional view of the environment and coarse ranges to objects in it. Using the coarse map, the system autonomously plans consecutive observations at the intersections of lines connecting object points, where the directions of the imaging are estimated easily and precisely. From two panoramic views at the two planned locations, a modified binocular stereo method yields a more precise, but with direction-dependent uncertainties, local map. New observation points are selected to decrease the uncertainty, and another local map is yielded, which is then integrated into a more reliable global representation of the world with the adjacent local maps     

Assessing stereo blindness and stereo acuity on digital displays

Previous studies on stereoscopic acuity have shown that the percentage of stereo blind subjects is relevant. Moreover, stereoscopic visualization is becoming widely diffused in different fields, like, e.g., entertainment, surgery or VR training, where it is necessary an accurate assessment of stereoscopic abilities of the involved subjects. Therefore, there might be the need of performing a stereo blindness and stereo acuity test before each visualization session involving stereoscopic images. In this paper, we propose a method to assess stereo acuity and stereo blindness directly on the chosen device, under the same visualization condition and setup adopted for the tasks to perform, in order to have the same perceptual response. We present software-based tests suitable for a generic stereoscopic displays, and we compare their effectiveness performing a comparison with a standard physical, card-based, test commonly used in assessment of stereo acuity and stereo blindness. We provide to the reader all the details to perform autonomously the tests, of which images will be downloadable from web.     

Rectified nearest feature line segment for pattern classification

This paper points out and analyzes the advantages and drawbacks of the nearest feature line (NFL) classifier. To overcome the shortcomings, a new feature subspace with two simple and effective improvements is built to represent each class. The proposed method, termed rectified nearest feature line segment (RNFLS), is shown to possess a novel property of concentration as a result of the added line segments (features), which significantly enhances the classification ability. Another remarkable merit is that RNFLS is applicable to complex tasks such as the two-spiral distribution, which the original NFL cannot deal with properly. Finally, experimental comparisons with NFL, NN(nearest neighbor), k-NN and NNL (nearest neighbor line) using both artificial and real-world data-sets demonstrate that RNFLS offers the best performance.     

Estimating parameters of noncentral catadioptric systems using bundle adjustment

This paper describes a new method to calibrate the intrinsic and extrinsic parameters of a generalized catadioptric camera (central or noncentral). The algorithm has two steps. The first one is the estimation of correspondences between incident lines in space and pixels (black box model calibration) in an arbitrary world reference frame. The second step is the calibration of the intrinsic parameters of the pinhole camera, the coefficients of the mirror expressed by a quadric (quadric mirror shape and the pose of the camera in relation to it), the position of the optical center of the camera in the world reference frame and its relative orientation (pose of the camera in world reference frame). A projection model relaxing Snell’s Law is derived. The deviations from Snell’s Law and the image reprojection errors are minimized by means of bundle adjustment. Information about the apparent contour of the mirror can be used to reduce the uncertainty in the estimation by introducing a new term in the cost function of the second step minimization process. Simulations and real experiments show good accuracy and robustness for this framework. However, the convergence is dependent on the initial guess as expected. A well-behaved algorithm to automatically generate the initial estimate to be used in the bundle adjustment is also presented.     

A new linear algorithm for calibrating central catadioptric cameras

In this paper, a novel linear calibration algorithm based on lines is presented for central catadioptric cameras. We firstly derive the relationship between the projection on the viewing sphere of a space point and its catadioptric image. And then by the relationship we establish a group of linear constraints on the catadioptric parameters from the catadioptric projections of spatial lines. By using these linear constraints, any central catadioptric camera can be fully calibrated from a single view of three or more lines without prior knowledge on the camera. Extensive experiments show this algorithm can improve the calibration's robustness.