Model based vehicle detection and tracking for autonomous urban driving

Situational awareness is crucial for autonomous driving in urban environments. This paper describes the moving vehicle detection and tracking module that we developed for our autonomous driving robot Junior. The robot won second place in the Urban Grand Challenge, an autonomous driving race organized by the U.S. Government in 2007. The module provides reliable detection and tracking of moving vehicles from a high-speed moving platform using laser range finders. Our approach models both dynamic and geometric properties of the tracked vehicles and estimates them using a single Bayes filter per vehicle. We present the notion of motion evidence, which allows us to overcome the low signal-to-noise ratio that arises during rapid detection of moving vehicles in noisy urban environments. Furthermore, we show how to build consistent and efficient 2D representations out of 3D range data and how to detect poorly visible black vehicles. Experimental validation includes the most challenging conditions presented at the Urban Grand Challenge as well as other urban settings.     

基于复合动态模型和证据融合架构的移动物体检测与跟踪方法

针对现有方法中移动物体检测与跟踪的准确性精度较低的缺点,提出一种基于多传感器检测分类的移动物体描述和感知方法：建立了一个包含核心对象动态特征和分类描述的复合模型,在此基础上设计了一个基于证据框架的信息感知与融合方法,通过整合动态模型和不确定性特征来实现对移动物体的检测和跟踪。为了验证所提方法的有效性,在一辆安装有雷达、激光雷达和摄像头的演示车上进行了相关实验,在不同驾驶场景下针对行人、卡车和轿车三个移动物体进行了检测与跟踪,实验结果证明所提方法具有非常高的准确性。     

Object tracking method based on data computing

The data computing process is utilized in various areas such as autonomous driving. Autonomous vehicles are intended to detect and track nearby moving objects avoiding collisions and to navigate in complex situations, such as heavy traffic and dense pedestrian areas. Therefore, object tracking is the core technology in the environment perception systems of autonomous vehicles and requires the monitoring of surrounding objects and the prediction of the moving states of objects in real time. In this paper, a multiple object tracking method based on light detection and ranging (LiDAR) data is proposed by using a Kalman filter and data computing process. We suppose that the movements of the tracking objects are captured consecutively as frames; thus, model-based detection and tracking of dynamic objects are possible. A Kalman filter is applied for predicting posterior state of tracking object based on anterior state of the tracking object. State denotes the positions, shapes, and sizes of objects. By computing the likelihood probability between predicted tracking objects and clusters which registered from tracking objects, the data association process of the tracking objects can be generated. Experimental results showed enhanced object tracking performance in a dynamic environment. The average matching probability of the tracking object was greater than 92.9%.

     

Driving Activity: How to Improve Knowledge of the Environment

This paper concerns the problem of driving assistance and, in particular, how to improve the perception of the surrounding environment to make this assistance really helpful. The main aims of a Driving Assistance System are to improve the security of the driver, passengers, and other road users. Driving is a complex activity, where the interactions between the driver, the vehicle, and the environment are continuous and numerous. The vehicle moves in a dynamic environment, so the Driving Assistance System, for its diagnosis, needs a map that represents as well as possible the actual situation of this environment. This paper presents a multi-sensor fusion module embedded in a real vehicle. The problem considered here is the dynamic reconstruction of the environment of the vehicle, based on measurements of a set of sensors.     

Tracking Multiple Moving Objects for Real-Time Robot Navigation

This paper proposes a method for detecting and tracking the motion of a large number of dynamic objects in crowded environments, such as concourses in railway stations or airports, shopping malls, or convention centers. With this motion information, a mobile vehicle is able to navigate autonomously among moving obstacles, operating at higher speeds and using more informed locomotion strategies that perform better than simple reactive manoeuvering strategies. Unlike many of the methods for motion detection and tracking discussed in the literature, our approach is not based on visual imagery but uses 2D range data obtained using a laser rangefinder. The direct availability of range information contributes to the real-time performance of our approach, which is a primary goal of the project, since the purpose of the vehicle is the transport of humans in crowded areas. Motion detection and tracking of dynamic objects is done by constructing a sequence of temporal lattice maps. These capture the time-varying nature of the environment, and are denoted as time-stamp maps. A time-stamp map is a projection of range information obtained over a short interval of time (a scan) onto a two-dimensional grid, where each cell which coincides with a specific range value is assigned a time stamp. Based on this representation, we devised two algorithms for motion detection and motion tracking. The approach is very efficient, with a complete cycle involving both motion detection and tracking taking 6 ms on a Pentium 166 MHz. The system has been demonstrated on an intelligent wheelchair operating in railway stations and convention centers during rush hour.     

Multi-spectral and multi-perspective video arrays for driver body tracking and activity analysis

This paper presents a novel approach to recognizing driver activities using a multi-perspective (i.e., four camera views) multi-modal (i.e., thermal infrared and color) video-based system for robust and real-time tracking of important body parts. The multi-perspective characteristics of the system provides redundant trajectories of the body parts, while the multi-modal characteristics of the system provides robustness and reliability of feature detection and tracking. The combination of a deterministic activity grammar (called ‘operation-triplet’) and a Hidden Markov model-based classifier provides semantic-level analysis of human activity. The application context for this research is that of intelligent vehicles and driver assistance systems. Experimental results in real-world street driving demonstrate effectiveness of the proposed system.     

基于卷积神经网络的驾驶辅助系统设计

为了减轻驾驶员在行驶过程中的操作负担,进而降低误差判断事件的出现几率,设计一种基于卷积神经网络的驾驶辅助系统。在执行良好的汽车导航架构中,限定Learning Navigation模块与Learning Controller模块的连接位置,再根据辅助驾驶传感器对于行驶画面的采集情况,对车辆巡航能力进行定向控制,抑制监测仪表中辅助波的过渡振动,完成驾驶辅助系统的需求与设计分析。在此基础上,确定辅助激活函数、约束仪表中的行车图像,建立标准化的卷积神经网络。按照驾驶辅助数据的学习结果,对其进行传输处理,进而连接驾驶辅助系统的Job请求,实现系统的顺利运行。利用卷积神经网络平台设计实车实验结果表明,应用驾驶辅助系统后,车辆监测仪表中辅助波振动幅度的最小值处于36-61Hz之间,平均波长偏移量明显减小,驾驶员的行驶操作负担得到有效缓解。     

On-road vehicle detection: a review

Developing on-board automotive driver assistance systems aiming to alert drivers about driving environments, and possible collision with other vehicles has attracted a lot of attention lately. In these systems, robust and reliable vehicle detection is a critical step. This paper presents a review of recent vision-based on-road vehicle detection systems. Our focus is on systems where the camera is mounted on the vehicle rather than being fixed such as in traffic/driveway monitoring systems. First, we discuss the problem of on-road vehicle detection using optical sensors followed by a brief review of intelligent vehicle research worldwide. Then, we discuss active and passive sensors to set the stage for vision-based vehicle detection. Methods aiming to quickly hypothesize the location of vehicles in an image as well as to verify the hypothesized locations are reviewed next. Integrating detection with tracking is also reviewed to illustrate the benefits of exploiting temporal continuity for vehicle detection. Finally, we present a critical overview of the methods discussed, we assess their potential for future deployment, and we present directions for future research.     

Real-time hierarchical stereo Visual SLAM in large-scale environments

In this paper we present a new real-time hierarchical (topological/metric) Visual SLAM system focusing on the localization of a vehicle in large-scale outdoor urban environments. It is exclusively based on the visual information provided by a cheap wide-angle stereo camera. Our approach divides the whole map into local sub-maps identified by the so-called fingerprints (vehicle poses). At the sub-map level (low level SLAM), 3D sequential mapping of natural landmarks and the robot location/orientation are obtained using a top-down Bayesian method to model the dynamic behavior. A higher topological level (high level SLAM) based on fingerprints has been added to reduce the global accumulated drift, keeping real-time constraints. Using this hierarchical strategy, we keep the local consistency of the metric sub-maps, by mean of the EKF, and global consistency by using the topological map and the MultiLevel Relaxation (MLR) algorithm. Some experimental results for different large-scale outdoor environments are presented, showing an almost constant processing time.     

Spatio-temporal analysis for obstacle detection in agricultural videos

Autonomous mobile vehicles are becoming commoner in outdoor scenarios for agricultural applications. They must be equipped with a robot navigation system for sensing, mapping, localization, path planning, and obstacle avoidance. In autonomous vehicles, safety becomes a major challenge where unexpected obstacles in the working area must be conveniently addressed. Of particular interest are, people or animals crossing in front of the vehicle or fixed/moving uncatalogued elements in specific positions. Detection of unexpected obstacles or elements on video sequences acquired with a machine vision system on-board a tractor moving in cornfields makes the main contribution to this research. We propose a new strategy for automatic video analysis to detect static/dynamic obstacles in agricultural environments via spatial-temporal analysis. At a first stage obstacles are detected by using spatial information based on spectral colour analysis and texture data. At a second stage temporal information is used to detect moving objects/obstacles at the scene, which is of particular interest in camouflaged elements within the environment. A main feature of our method is that it does not require any training process. Another feature of our approach consists in the spatial analysis to obtain an initial segmentation of interesting objects; afterwards, temporal information is used for discriminating between moving and static objects. To the best of our knowledge in the field of agricultural image analysis, classical approaches make use of either spatial or temporal information, but not both at the same time, making an important contribution. Our method shows favourable results when tested in different outdoor scenarios in agricultural environments, which are really complex, mainly due to the high variability in the illumination conditions, causing undesired effects such as shadows and alternating lighted and dark areas. Dynamic background, camera vibrations and static and dynamic objects are also factors complicating the situation. The results are comparable to those obtained with other state-of-art techniques reported in literature.     

A Computer Vision System for Lateral Localization

For urban driving, knowledge of ego‐vehicle's position is a critical piece of information that enables advanced driver‐assistance systems or self‐driving cars to execute safety‐related, autonomous driving maneuvers. This is because, without knowing the current location, it is very hard to autonomously execute any driving maneuvers for the future. The existing solutions for localization rely on a combination of a Global Navigation Satellite System, an inertial measurement unit, and a digital map. However, in urban driving environments, due to poor satellite geometry and disruption of radio signal reception, their longitudinal and lateral errors are too significant to be used for an autonomous system. To enhance the existing system's localization capability, this work presents an effort to develop a vision‐based lateral localization algorithm. The algorithm aims at reliably counting, with or without observations of lane‐markings, the number of road‐lanes and identifying the index of the road‐lane on the roadway upon which our vehicle happens to be driving. Tests of the proposed algorithms against intercity and interstate highway videos showed promising results in terms of counting the number of road‐lanes and the indices of the current road‐lanes.     

基于运动模型的道路识别与跟踪算法的研究

自主驾驶与辅助导航是目前国际上研究的热问题,通过对室外行驶车辆上的ＣＣＤ摄像机所采集的长序列立体图象的处理与分析,研究公路汽车自动视觉导航中的道路识别与跟踪问题。     

Calibration of double stripe 3D laser scanner systems using planarity and orthogonality constraints

In this study, 3D scanning systems that utilize a pair of laser stripes are studied. Three types of scanning systems are implemented to scan environments, rough surfaces of near planar objects and small 3D objects. These scanners make use of double laser stripes to minimize the undesired effect of occlusions. Calibration of these scanning systems is crucially important for the alignment of 3D points which are reconstructed from different stripes. In this paper, the main focus is on the calibration problem, following a treatment on the pre-processing of stripe projections using dynamic programming and localization of 2D image points with sub-pixel accuracy. The 3D points corresponding to laser stripes are used in an optimization procedure that imposes geometrical constraints such as coplanarities and orthogonalities. It is shown that, calibration procedure proposed here, significantly improves the alignment of 3D points scanned using two laser stripes.     

Pedestrian tracking by fusion of thermal-visible surveillance videos

In this paper we introduce a system to track pedestrians using a combined input from RGB and thermal cameras. Two major contributions are presented here. First is the novel probabilistic model of the scene background where each pixel is represented as a multi-modal distribution with the changing number of modalities for both color and thermal input. We demonstrate how to eliminate the influence of shadows with this type of fusion. Second, based on our background model we introduce a pedestrian tracker designed as a particle filter. We further develop a number of informed reversible transformations to sample the model probability space in order to maximize our model posterior probability. The novelty of our tracking approach also comes from a way we formulate observation likelihoods to account for 3D locations of the bodies with respect to the camera and occlusions by other tracked human bodies as well as static objects. The results of tracking on color and thermal sequences demonstrate that our algorithm is robust to illumination noise and performs well in the outdoor environments.     

基于稀疏帧检测的交通目标跟踪

为了获取高速公路交通视频中目标车辆的行驶轨迹,提出一种基于视频的多目标车辆跟踪及实时轨迹分布算法,为交通管理系统和交通决策提供目标车辆交通信息.首先,使用YOLOv4算法检测目标车辆位置及置信度.其次,在不同场景条件下,使用提出的基于稀疏帧检测的跟踪方法,结合KCF跟踪算法,将车辆数据进行关联获取完整轨迹.最后,用车辆分布图和交通场景俯视图显示轨迹,便于交通管理与分析.实验结果表明,提出的跟踪方法在车辆跟踪中有较高的跟踪正确率,同时基于稀疏帧检测的跟踪方法处理速度也较快,实时轨迹分布正确反映了真实场景的车道信息以及目标车辆运动信息.     

The ergonomic value of a bidirectional haptic interface when driving a highly automated vehicle

Advances in technology have fueled the development of driver assistance systems. Even today, these systems can take over parts of the driving task. However, the interface becomes more and more complex with an increasing number of functions. One way to reduce such complexity is to venture the haptic channel. While haptic feedback in lateral direction is comparatively easy to realize via the steering wheel, the longitudinal direction forms a challenge. With conventional control elements, that is, pedals, haptic interaction can only be partially realized (this is due to the division of accelerator and brake pedals). Haptic signals, like forces added to the accelerator pedal, can only transmit information regarding the amount of acceleration, not the desired deceleration. In this context, two-dimensional control elements show great potential regarding future highly automated vehicle driving. Therefore, an experiment conducted at the Institute of Ergonomics of the Technische Universität München investigated the influence of haptic feedback of assistance systems on driving performance when using an active side stick as control element. Additionally, the impact of vehicle vibrations and accelerations were explored. Besides objective performance data, subjective assessment was also reported. The results show that adding assistance significantly improves driving performance. Moreover, subjective ratings indicate a reduction in workload. Accelerations and vibrations, however, had no verifiable effect on the driving performance. This fact was confirmed by the subjects’ subjective assessment. This paper shows that two-dimensional control elements can be a reasonable alternative to steering wheel and pedals when driving a highly automated vehicle.     

动态环境中多帧点云融合算法及三维目标检测算法研究

低线束激光雷达扫描的点云数据较为稀疏,导致无人驾驶环境感知系统中三维目标检测效果欠佳,通过多帧点云配准可实现稀疏点云稠密化,但动态环境中的行人与移动车辆会降低激光雷达的定位精度,也会造成融合帧中运动目标上的点云偏移较大。针对上述问题,提出了一种动态环境中多帧点云融合算法,利用该算法在园区道路实况下进行三维目标检测,提高了低线束激光雷达的三维目标检测精度。利用16线和40线激光雷达采集的行驶路况数据进行实验,结果表明该算法能够增强稀疏点云密度,改善低成本激光雷达的环境感知能力。     

Robot-vision architecture for real-time 6-DOF object localization

This paper presents a new robot-vision system architecture for real-time moving object localization. The 6-DOF (3 translation and 3 rotation) motion of the objects is detected and tracked accurately in clutter using a model-based approach without information of the objects’ initial positions. An object identification task and an object tracking task are combined under this architecture. The computational time-lag between the two tasks is absorbed by a large amount of frame memory. The tasks are implemented as independent software modules using stereo-vision-based methods which can deal with objects of various shapes with edges, including planar to smooth-curved objects, in cluttered environments. This architecture also leads to failure-recoverable object tracking, because the tracking processes can be automatically recovered, even if the moving objects are lost while tracking. Experimental results obtained with prototype systems demonstrate the effectiveness of the proposed architecture.     

Low cost IMU–Odometer–GPS ego localization for unusual maneuvers

This paper presents the problem of outdoor vehicle localization during unusual maneuvers with the Interacting Multiple Model (IMM) and Extended Kalman Filter (EKF) approaches. IMM, contrary to classical methods, is based on the discretization of the vehicle evolution space into simple maneuvers. Each maneuver is represented by a simple dynamic model such as a constant velocity or a constant turning model. This allows the method to be optimized for highly dynamic vehicles. In this work, we focus on unusual vehicle maneuvers during some special driving situations, including very strong accelerations, high speed turnings or backwards driving with stop stages. The presented results are based on real measurements collected from different scenarios. Based on an EKF vs. IMM comparison, these results show a real interest of using the IMM method in order to take into account unusual maneuvers.     

How the Location of the Range Sensor Affects EKF-based Localization

The article proposes a solution to map-based self-localization for an autonomous robot operating in cluttered and crowded environments. To detect features for localization, 2D laser range-finders traditionally scan a plane parallel to the floor. This work hypothesizes the existence of a ??low frequency cross-section?? of the 3D Workspace where cluttered and dynamic environments become ??more regular?? and ??less dynamic??. The contribution of the article is twofold. First, an ??unevenness index?? U is introduced to quantitatively measure the complexity of the environment as it would be perceived if the laser range-finder were located at different heights from the floor. The article shows that, by choosing the laser scanning plane to statistically minimize U (in most cases, above the heads of people), it is possible to deal more efficiently with non-linearities in the measurement model, moving objects and occluded features. Second, it is demonstrated that, when adopting an extended Kalman filter for position tracking (a very common and widely used technique in real-world scenarios), the a posteriori covariance of the estimated robot pose converges faster, on average, when U is lower, which leads to better localization performance. Experimental results show hours of continuous robot operation in real-world, cluttered and crowded environments.