Real-time people localization and tracking through fixed stereo vision

Detecting, locating, and tracking people in a dynamic environment is important in many applications, ranging from security and environmental surveillance to assistance to people in domestic environments, to the analysis of human activities. To this end, several methods for tracking people have been developed in the field of Computer Vision using different settings, such as monocular cameras, stereo sensors, multiple cameras. In this article we describe a method for People Localization and Tracking (PLT) based on a calibrated fixed stereo vision sensor, its implementation and experimental results. The system analyzes three components of the stereo data (the left intensity image, the disparity image, and the 3-D world locations of measured points) to dynamically update a model of the background; extract foreground objects, such as people and rearranged furniture; track their positions in the world. The system is mostly suitable for indoor medium size environments. It can reliably detect and track people moving in an medium size area (a room or a corridor) in front of the sensor with high reliability and good precision.     

Pedestrian detection by means of far-infrared stereo vision

This article presents a stereo system for the detection of pedestrians using far-infrared cameras. Since pedestrian detection in far-infrared images can be difficult in some environmental conditions, the system exploits three different detection approaches: warm area detection, edge-based detection, and disparity computation. A final validation process is performed using head morphological and thermal characteristics. Currently, neither temporal correlation, nor motion cues are used in this processing.The developed system has been implemented on an experimental vehicle equipped with two infrared cameras and preliminarily tested in different situations.     

Building facade detection,segmentation, and parameter estimation for mobile robot stereo vision

Building facade detection is an important problem in computer vision, with applications in mobile robotics and semantic scene understanding. In particular, mobile platform localization and guidance in urban environments can be enabled with accurate models of the various building facades in a scene. Toward that end, we present a system for detection, segmentation, and parameter estimation of building facades in stereo imagery. The proposed method incorporates multilevel appearance and disparity features in a binary discriminative model, and generates a set of candidate planes by sampling and clustering points from the image with Random Sample Consensus (RANSAC), using local normal estimates derived from Principal Component Analysis (PCA) to inform the planar models. These two models are incorporated into a two-layer Markov Random Field (MRF): an appearance- and disparity-based discriminative classifier at the mid-level, and a geometric model to segment the building pixels into facades at the high-level. By using object-specific stereo features, our discriminative classifier is able to achieve substantially higher accuracy than standard boosting or modeling with only appearance-based features. Furthermore, the results of our MRF classification indicate a strong improvement in accuracy for the binary building detection problem and the labeled planar surface models provide a good approximation to the ground truth planes.     

Accurate hardware-based stereo vision

To enable both accurate and fast real-time stereo vision in embedded systems, we propose a novel stereo matching algorithm that is designed for high efficiency when realized in hardware. We evaluate its accuracy using the Middlebury Stereo Evaluation, revealing its high performance at minimum tolerance. To outline the resource efficiency of the algorithm, we present its realization as an Intellectual Property (IP) core that is designed for the deployment in Field Programmable Gate Arrays (FPGAs) and Application Specific Integrated Circuits (ASICs).     

基于双目视觉的无人飞行器目标跟踪与定位

针对小型无人飞行器跟踪目标的问题,提出了一种基于双目视觉和Camshift算法的无人飞行器目标跟踪以及定位算法。双目相机得到的左右图像通过Camshift算法处理可得到目标中心特征点,对目标中心特征点进行三维重建,得到机体坐标系下无人飞行器与目标间的相对位置和偏航角,应用卡尔曼滤波算法对测量值进行了优化,将所得估计值作为飞行控制系统的反馈输入值,实现了无人飞行器自主跟踪飞行。结果表明所提算法误差较小,具有较高的稳定性与精确性。     

Speech recognition using a stereo vision neural network model

This paper describes a new algorithm for speech recognition by using stereo vision pattern recognition equations with competition and cooperation. In our research, we applied recently developed 3-layered neural net (3LNN) equations to speech recognition. Our proposed acoustic models using these equations yield better recognition results than the hidden Markov model (HMM). When using a 216 (240) word database, stereo vision acoustic models gave 6.5% (6.6%) higher accuracy than HMMs. This work was presented, in part, at the Fourth International Symposium on Artificial Life and Robotics, Oita, Japan, January 19–22, 1999     

3D soft-tissue tracking using spatial-color joint probability distribution and thin-plate spline model

Visual tracking techniques based on stereo endoscope are developed to measure tissue motion in robot-assisted minimally invasive surgery. However, accurate 3D tracking of tissue surfaces remains challenging due to complicated deformation, poor imaging conditions, specular reflections and other dynamic effects during surgery. This study employs a robust and efficient 3D tracking scheme with two independent recursive processes, namely kernel-based inter-frame motion estimation and model-based intra-frame 3D matching. In the first process, target region is represented in joint spatial-color space for robust estimation. By defining a probabilistic similarity measure, a mean-shift-based iterative algorithm is derived for location of the target region in a new image. In the second process, the thin-plate spline model is used to fit the 3D shape of tissue surfaces around the target region. An iterative algorithm based on an efficient second-order minimization technique is derived to compute optimal model parameters. The two processes can be computed in parallel. Their outputs are combined to recover 3D information about the target region. The performance of the proposed method is validated using phantom heart videos and in vivo videos acquired by the daVinci^®${\mathrm{daVinci}}^{\circledR }$ surgical robotic platform and a synthesized data set with known ground truth.     

Coarse-to-fine stereo vision with accurate 3D boundaries

This paper presents methods for efficient recovery of accurate binocular disparity estimates in the vicinity of 3D surface discontinuities. Of particular concern are methods that impact coarse-to-fine, local block-based matching as it forms the basis of the fastest and the most resource efficient stereo computation procedures. A novel coarse-to-fine refinement procedure that adapts match window support across scale to ameliorate corruption of disparity estimates near boundaries is presented. Extensions are included to account for half-occlusions and colour uniformity. Empirical results show that incorporation of these advances in the standard coarse-to-fine, block matching framework reduces disparity errors by more than a factor of two, while performing little extra computation, preserving low complexity and the parallel/pipeline nature of the framework. Moreover, the proposed advances prove to be beneficial for CTF global matchers as well.     

一种基于立体视觉的运动目标检测算法

在目标检测中采用立体视觉方法。首先对立体图像对进行匹配求取场景的视差图,再运用基于视差的背景差分法获得含有运动目标的前景区域,最后根据前景区域的视差和位置分布准确定位各运动目标。立体视觉方法有效解决了单目视觉检测方法中的一些难点问题,可以克服光线的变化和阴影干扰对目标检测带来的影响,在多个目标发生部分遮挡时仍能正确区分各运动目标。     

基于机器视觉的打磨工件跟踪方法研究

为克服在自动化生产中跟踪打磨工件时受尺度变化、部分遮挡等因素影响跟踪效果的问题,提出了一种基于机器视觉的打磨工件长时间目标跟踪算法。该算法对传统的KCF目标跟踪算法做了尺度估计、质心位置预测和质心修正重定位的改进,首先在与传统KCF算法结合的基础上,计算出运动目标质心位置的同时引进尺度金字塔来进行目标尺度估计;然后提出一种质心位置预测方法估计其质心运动;最后为了防止目标丢失使用质心修正重定位方法进行质心重定位,提高了跟踪的稳定性。实验证明,在工件发生尺度变化、部分遮挡等的情况下,本文方法具有良好的检测效果,并具有较强的鲁棒性。     

Rover navigation using stereo ego-motion

Robust navigation for mobile robots over long distances requires an accurate method for tracking the robot position in the environment. Promising techniques for position estimation by determining the camera ego-motion from monocular or stereo sequences have been previously described. However, long-distance navigation requires both a high level of robustness and a low rate of error growth. In this paper, we describe a methodology for long-distance rover navigation that meets these goals using robust estimation of ego-motion. The basic method is a maximum-likelihood ego-motion algorithm that models the error in stereo matching as a normal distribution elongated along the (parallel) camera viewing axes. Several mechanisms are described for improving navigation robustness in the context of this methodology. In addition, we show that a system based on only camera ego-motion estimates will accumulate errors with super-linear growth in the distance traveled, owing to increasing orientation errors. When an absolute orientation sensor is incorporated, the error growth can be reduced to a linear function of the distance traveled. We have tested these techniques using both extensive simulation and hundreds of real rover images and have achieved a low, linear rate of error growth. This method has been implemented to run on-board a prototype Mars rover.     

Three dimentional measurement of fish movement using stereo vision

Recently, the technology of cultivation has been and continuous to be developed in the fisheries and the studies for a high production efficiency are widely conducted. The reaction of fishes under the sound should be studied in order to develop the fish tank with less factors of stress. In this paper, three-dimensional measurement system of the fish motion has been presented in order to study the relation of the fishes under the stimulating sound. Three kinds of fishes are used to detect the reaction against the sounds. Frequency analysis is executed by using the threedimensional position data. This work was presented in part at the 13th International Symposium on Artificial Life and Robotics, Oita, Japan, January 31–February 2, 2008     

Similarity measures for image matching despite occlusions in stereo vision

In the context of computer vision, matching can be done with similarity measures. This paper presents the classification of these measures into five families. In addition, 18 measures based on robust statistics, previously proposed in order to deal with the problem of occlusions, are studied and compared to the state of the art. A new evaluation protocol and new analyses are proposed and the results highlight the most efficient measures, first, near occlusions, the smooth median powered deviation, and second, near discontinuities, a non-parametric transform-based measure, CENSUS.     

A multiview 3D modeling system based on stereo vision techniques

This paper introduces a stereo vision system to automatically generate 3D models of real objects. 3D model generation is based on the merging of multiview range images obtained from a digital stereo camera. Stereo images obtained from the camera are rectified, and a correlation-based stereo matching technique reconstructs range images from them. A turntable stage is also employed to obtain multiple range images of the objects. To register range images into a common coordinate system automatically, we introduce and calibrate a turntable coordinate system with respect to the camera coordinate system. After the registration of multiview range images, a 3D model is reconstructed using a volumetric integration technique. Error analysis on turntable calibration and 3D model reconstruction shows the accuracy of our 3D modeling system.Received: 2 August 2003, Accepted: 20 September 2004, Published online: 25 February 2005 Correspondence to: S.Y. Park     

Automatic feature matching in uncalibrated stereo vision through the use of color

One difficult problem in stereo vision is how to establish correspondence between features extracted from a pair of images. The difficulty is due to ambiguities or inconsistencies of available information on images. In this paper, we invetigate stereo correspondence problem in the framework of color stereo vision. We propose the use of a matching consistence (MC) constraint in RGB color space and the generalized epipolar geometry to develop an automatic feature matching algorithm.     

A Bayesian plan-view map based approach for multiple-person detection and tracking

This work proposes a novel approach for people detection and tracking in colour-with-depth sequences using a particle filtering approach. Detection and tracking are performed in plan-view maps integrating occupancy and height information with a novel plan-view map representation for colour information. Using the three maps, we propose a multiple particle filtering algorithm for people detection and tracking. The observation model proposed integrates information from the three maps so that people with different coloured clothes are not confused even when they interact at close distances. To avoid the coalescence problem when people with similar coloured clothes approach each other, the weight of particles is modified by an interaction factor that combines colour and position information. The algorithm also avoids the coalescence problem in case of total occlusion by means of an occlusion detection and recovering mechanism. Finally, a solution is proposed to improve the exponential complexity of multiple particle filters so that the algorithm proposed has linear complexity.The approach proposed has been tested in several colour-with-depth sequences where people move and interact freely in the environment. In the sequences, people walk at different distances, cross their paths causing frequent occlusions, jump, run and have close interactions such as shaking hands or embracing each other. The experimental results show that our proposal is able to detect and keep track of every person with a low error and deals with partial and total occlusions. Besides, the detection and tracking techniques presented are appropriate for large tracking problems in real-time applications since their complexity is linear, are suitable for parallel processing and allow the integration of information provided by multiple stereo vision sensors.     

3-D shape reconstruction in an active stereo vision system using genetic algorithms

The recovery of 3-D shape information (depth) using stereo vision analysis is one of the major areas in computer vision and has given rise to a great deal of literature in the recent past. The widely known stereo vision methods are the passive stereo vision approaches that use two cameras. Obtaining 3-D information involves the identification of the corresponding 2-D points between left and right images. Most existing methods tackle this matching task from singular points, i.e. finding points in both image planes with more or less the same neighborhood characteristics. One key problem we have to solve is that we are on the first instance unable to know a priori whether a point in the first image has a correspondence or not due to surface occlusion or simply because it has been projected out of the scope of the second camera. This makes the matching process very difficult and imposes a need of an a posteriori stage to remove false matching.In this paper we are concerned with the active stereo vision systems which offer an alternative to the passive stereo vision systems. In our system, a light projector that illuminates objects to be analyzed by a pyramid-shaped laser beam replaces one of the two cameras. The projections of laser rays on the objects are detected as spots in the image. In this particular case, only one image needs to be treated, and the stereo matching problem boils down to associating the laser rays and their corresponding real spots in the 2-D image. We have expressed this problem as a minimization of a global function that we propose to perform using Genetic Algorithms (GAs). We have implemented two different algorithms: in the first, GAs are performed after a deterministic search. In the second, data is partitioned into clusters and GAs are independently applied in each cluster. In our second contribution in this paper, we have described an efficient system calibration method. Experimental results are presented to illustrate the feasibility of our approach. The proposed method yields high accuracy 3-D reconstruction even for complex objects. We conclude that GAs can effectively be applied to this matching problem.     

提高频率域可分级视频信源编码效率的研究

在ＭＰＥＧ－２视频编码标准的框架基础上，本文引入了一种提高频率域可分级视频编码效率的方法。通过在８×８ＤＣＴ块中合理地分割系数及尽量减少在低分辨率层中所必需传送的块结束码（ＥＯＢ）的数量，使为获得频率域可分级性所付出的代价明显减小。计算机模拟实验证实了采用该方法可提高编码效率     

基于双目视觉的运动目标跟踪与测量

在研究融合运动目标位置预测的Mean Shift跟踪算法和双目立体视觉中的空间点定位算法的基础上,基于双目视觉设计了双目立体视觉运动目标跟踪和测量系统,并在所进行的跟踪与测量实验中,提取了运动目标质心的三维坐标序列,实现了对目标深度和速度的测量。     

Evaluation of stereo correspondence algorithms and their implementation on FPGA

The accuracy of stereo vision has been considerably improved in the last decade, but real-time stereo matching is still a challenge for embedded systems where the limited resources do not permit fast operation of sophisticated approaches. This work presents an evaluation of area-based algorithms used for calculating distance in stereoscopic vision systems, their hardware architectures for implementation on FPGA and the cost of their accuracies in terms of FPGA hardware resources. The results show the trade-off between the quality of such maps and the hardware resources which each solution demands, so they serve as a guide for implementing stereo correspondence algorithms in real-time processing systems.