A real-time global stereo-matching on FPGA

An improved global stereo matching algorithm is implemented on a single FPGA for real-time applications. Stereo matching is widely used in stereo vision systems, i.e. objects detection and autonomous vehicles. Global algorithms have much more accurate results than local algorithms, but global algorithms are not implemented on FPGA since they rely over high-end hardware resources. In this implementation the stereo pairs are divided into blocks, the hardware resources are reduced by processing one block once. The hardware implementation is based on a Xilinx Kintex 7 FPGA. Experiment results show that the proposed implementation has an accurate result for the Middlebury benchmarks and 30 frames per second (fps) @1920 × 1680 is achieved.     

基于FPGA的双目立体视觉系统

立体视觉的目的之一就是为了获得周围场景的3维信息,其关键在于匹配算法。然而即便是使用目前先进的通用处理器,其计算致密视差图所需的时间仍无法满足高速自主导航的需求。为了解决这个问题,提出了一种基于现场可编程门阵列(FPGA)的双目立体视觉系统的设计方案,同时介绍了系统的硬件结构,并在讨论区域匹配的快速算法的基础上,提出了基于FPGA的像素序列和并行窗口算法框架,用以实现零均值像素灰度差平方和(ZSSD)的匹配算法。该算法是先将视频信号经解码芯片生成场景立体图像对,并由FPGA来完成立体图像对的几何校正和ZSSD匹配算法,然后将获得的致密视差图通过PC I总线发送至上位机。实践表明,该算法效果好、速度快,不仅具有较强的鲁棒性,并且硬件系统性能稳定、可靠。此外,该方案还适用于像素灰度差的绝对值和(SAD)和像素灰度差的平方和(SSD)等多种传统区域匹配算法的快速实现和实时处理。     

An FPGA stereo matching unit based on fuzzy logic

Stereo matching is one of the most used algorithms in real-time image processing applications such as positioning systems for mobile robots, three-dimensional building mapping and recognition, detection and three-dimensional reconstruction of objects. In order to improve the performance, stereo matching algorithms often have been implemented in dedicated hardware such as FPGA or GPU devices. In this paper an FPGA stereo matching unit based on fuzzy logic is described. The proposed algorithm consists of three stages. First, three similarity parameters inherent to each pixel contained in the input stereo pair are computed. Then, the similarity parameters are sent to a fuzzy inference system which determines a fuzzy-similarity value. Finally, the disparity value is defined as the index which maximizes the fuzzy-similarity values (zero up to d_max). Dense disparity maps are computed at a rate of 76 frames per second for input stereo pairs of 1280 × 1024 pixel resolution and a maximum expected disparity equal to 15. The developed FPGA architecture provides reduction of the hardware resource demand compared to other FPGA-based stereo matching algorithms: near to 72.35% for logic units and near to 32.24% for bits of memory. In addition, the developed FPGA architecture increases the processing speed: near to 34.90% pixels per second and outperforms the accuracy of most of real-time stereo matching algorithms in the state of the art.     

FPGA based disparity map computation with vergence control

Depth estimation in a scene using image pairs acquired by a stereo camera setup, is one of the important tasks of stereo vision systems. The disparity between the stereo images allows for 3D information acquisition which is indispensable in many machine vision applications. Practical stereo vision systems involve wide ranges of disparity levels. Considering that disparity map extraction of an image is a computationally demanding task, practical real-time FPGA based algorithms require increased device utilization resource usage, depending on the disparity levels operational range, which leads to significant power consumption. In this paper a new hardware-efficient real-time disparity map computation module is developed. The module constantly estimates the precisely required range of disparity levels upon a given stereo image set, maintaining this range as low as possible by verging the stereo setup cameras axes. This enables a parallel-pipelined design, for the overall module, realized on a single FPGA device of the Altera Stratix IV family. Accurate disparity maps are computed at a rate of more than 320 frames per second, for a stereo image pair of 640 × 480 pixels spatial resolution with a disparity range of 80 pixels. The presented technique provides very good processing speed at the expense of accuracy, with very good scalability in terms of disparity levels. The proposed method enables a suitable module delivering high performance in real-time stereo vision applications, where space and power are significant concerns.     

FPGA based real-time on-road stereo vision system

Accurate and Real-Time stereo vision is an essential need for many computer vision applications, such as On-Road stereo vision system. In this paper, a fast and accurate system for On-Road stereo vision application is presented. In order to achieve this purpose, first, an algorithm is presented that is optimized for hardware implementation and On-Road application, then an appropriate hardware architecture for this algorithm is proposed. The approach uses gradient and Census transform as initial cost function, then cost is aggregate in cross-based supported reign. LR-check and disparity refinement is also utilized to improve final disparity. The proposed design is a standalone stereo vision system where all steps of algorithm is implemented on a hardware platform. Overall system is implemented on FPGA platform and it is tested on KITTI Database. The proposed system is also tested on Middlebury database without any changes in parameters. Experimental results show that the proposed system has high accuracy in KITTI and Middlebury database. In term of hardware resource, the system occupy %%66 of XC6VLX240T FPGA device for 1920 × 1080 image with 96 disparity levels that operated at 53.5 Fps.     

A real-time fuzzy hardware structure for disparity map computation

Stereo images acquired by a stereo camera setup provide depth estimation of a scene. Numerous machine vision applications deal with retrieval of 3D information. Disparity map recovery from a stereo image pair involves computationally complex algorithms. Previous methods of disparity map computation are mainly restricted to software-based techniques on general-purpose architectures, presenting relatively high execution time. In this paper, a new hardware-implemented real-time disparity map computation module is realized. This enables a hardware-based fuzzy inference system parallel-pipelined design, for the overall module, implemented on a single FPGA device with a typical operating frequency of 138 MHz. This provides accurate disparity map computation at a rate of nearly 440 frames per second, given a stereo image pair with a disparity range of 80 pixels and 640 × 480 pixels spatial resolution. The proposed method allows a fast disparity map computational module to be built, enabling a suitable module for real-time stereo vision applications.     

基于FPGA的视觉处理系统设计与实现

从计算机视觉系统的基本体系结构出发,指出了计算机视觉系统要实现的主要功能,从理论上探讨了计算机视觉系统在硬件实现层次上存在的问题.进一步以立体视觉的应用要求为例提出了以现场可编程门阵列(FPGA)为核心芯片的视觉处理系统.因为FPGA具有极强的可重构性,可承担部分原来由上位机软件完成的运算,增强了视觉处理的实时性.其应用于计算机三维立体视觉系统中,作为前端的图像采集器和视觉协处理器取得了良好的效果.     

FPGA上基于Hausdorff距离的图像匹配并行算法设计与实现

基于Hausdorff距离的图像匹配算法鲁棒性较好,但计算代价较大,软件实现方案很难满足实时性要求。为了解决这个问题,本文在基于局部Hausdorff距离的图像匹配算法基础上提出了一种鲁棒而实时的FPGA实现方案。为了充分有效利用FPGA的硬件资源,首先对传统串行算法进行并行性分析,提出了一个并行算法;然后以此为基础设计了一种三段式粗粒度流水体系结构,并将其映射到FPGA上进行实现。实验结果表明,该系统在性能上优于其它相关工作,与PC(Pentium42.8GHz)上的软件实现方案相比可以达到接近50倍的加速比。     

A Performance Study on Different Cost Aggregation Approaches Used in Real-Time Stereo Matching

Many vision applications require high-accuracy dense disparity maps in real-time and online. Due to time constraint, most real-time stereo applications rely on local winner-takes-all optimization in the disparity computation process. These local approaches are generally outperformed by offline global optimization based algorithms. However, recent research shows that, through carefully selecting and aggregating the matching costs of neighboring pixels, the disparity maps produced by a local approach can be more accurate than those generated by many global optimization techniques. We are therefore motivated to investigate whether these cost aggregation approaches can be adopted in real-time stereo applications and, if so, how well they perform under the real-time constraint. The evaluation is conducted on a real-time stereo platform, which utilizes the processing power of programmable graphics hardware. Six recent cost aggregation approaches are implemented and optimized for graphics hardware so that real-time speed can be achieved. The performances of these aggregation approaches in terms of both processing speed and result quality are reported.     

Real-Time Correlation-Based Stereo Vision with Reduced Border Errors

This paper describes a real-time stereo vision system that is required to support high-level object based tasks in a tele-operated environment. Stereo vision is computationally expensive, due to having to find corresponding pixels. Correlation is a fast, standard way to solve the correspondence problem. This paper analyses the behaviour of correlation based stereo to find ways to improve its quality while maintaining its real-time suitability. Three methods are suggested. Two of them aim to improve the disparity image especially at depth discontinuities, while one targets the identification of possible errors in general. Results are given on real stereo images with ground truth. A comparison with five standard correlation methods is provided. All proposed algorithms are described in detail and performance issues and optimisation are discussed. Finally, performance results of individual parts of the stereo algorithm are shown, including rectification, filtering andcorrelation using all proposed methods. The implemented system shows that errors of simple stereo correlation, especially in object border regions, can be reduced in real-time using non-specialised computer hardware.     

A method of parallel-sequential detection of image features in computer vision systems

The paper is devoted to the problem of formalization of software-hardware solutions in designing real-time computer vision systems. The main attention is paid to the methods of implementation of low-level operations that find features (simple elements) in the image input to the system. Algorithmic types of detectors of simple elements in images are analyzed from the point of view of hardware organization in computer vision systems. In this connection, the necessary performance and memory resources are estimated. The capabilities of parallel and pipeline execution of detector algorithms are investigated. A method of using a field programmable gate array and a digital signal processor in solving the problem of image processing in real-time computer vision systems is considered in detail.     

Fuzzy logic-based embedded system for video de-interlacing

Video de-interlacing algorithms perform a crucial task in video processing. Despite these algorithms are developed using software implementations, their implementations in hardware are required to achieve real-time operation. This paper describes the development of an embedded system for video de-interlacing. The algorithm for video de-interlacing uses three fuzzy logic-based systems to tackle three relevant features in video sequences: motion, edges, and picture repetition. The proposed strategy implements the algorithm as a hardware IP core on a FPGA-based embedded system. The paper details the proposed architecture and the design methodology to develop it. The resulting embedded system is verified on a FPGA development board and it is able to de-interlace in real-time.     

一种道路识别算法的硬件设计与实现

车辆视觉导航算法的硬件实现具有实际意义,是目前的研究热点之一;为克服传统算法硬件设计实现比较复杂、调试困难、对设计人员要求较高等缺点,对基于高级语言的复杂算法硬件设计实现方法进行了研究;分析了基于Handel-C语言的道路识别算法FPGA硬件设计与实现过程,并进行了实验验证;实验结果表明,和目前采用的VHDL语言等设计方法相比,该方法具有设计灵活、开发周期短、资源利用合理等优点,同时易于软硬件协同设计.     

Accurate hardware-based stereo vision

To enable both accurate and fast real-time stereo vision in embedded systems, we propose a novel stereo matching algorithm that is designed for high efficiency when realized in hardware. We evaluate its accuracy using the Middlebury Stereo Evaluation, revealing its high performance at minimum tolerance. To outline the resource efficiency of the algorithm, we present its realization as an Intellectual Property (IP) core that is designed for the deployment in Field Programmable Gate Arrays (FPGAs) and Application Specific Integrated Circuits (ASICs).     

一种改进的快速立体匹配方法

首先介绍了立体匹配的一般步骤,再考虑到立体匹配的实时性,从众多立体匹配算法中选择了基于区域的WTA相关匹配算法。同时,为了保证匹配的正确性,降低误匹配率,本文在立体匹配中采用了亚像素插值和边缘检测等多项改进措施。实验结果表明,本文的立体匹配算法既具有良好的实时性也具有较高的精度,完全能满足移动机器人双目立体视觉等应用要求。     

基于FPGA的H．264变换量化的高性能的硬件实现

针对H．264视频标准中一个功能频繁调用的变换量化模块，提出了一种高性能的FPGA硬件实现方法。并完成了其硬件原型的设计。该硬件原型包含了从残差形成到宏块重建的变换量化全过程。其可以构成DSP的协处理器，用于完成H．264实时编解码。该硬件原型根据算法特点和数据流特点，采用了流水线控制策略和分时复用技术，同时合理利用FPGA片内资源，从而提高了系统性能。仿真结果表明。该设计能满足高清数字视频的实时处理应用。     

Fast buffering for FPGA implementation of vision-based object recognition systems

Real-time frame rate is an important factor for practical deployment of computer vision systems. Field programmable gate array (FPGA) technology has been considered for many applications due to its parallel computing capability. FPGA implementations of computer vision algorithms normally involve buffering data on external memory devices, which could slow down the whole system. This paper proposes a buffering scheme suitable for implementing real-time vision-based systems on an FPGA that does not require external memory to buffer data. A stop sign detection system implemented on an FPGA employing the proposed buffering scheme is presented as an example system. This system is capable of processing over 200?fps at the frame size of 480?×?752 pixels.     

A reconfigurable architecture for autonomous visual-navigation

Abstract. This paper describes the design of a reconfigurable architecture for implementing image processing algorithms. This architecture is a pipeline of small identical processing elements that contain a programmable logic device (FPGA) and double port memories. This processing system has been adapted to accelerate the computation of differential algorithms. The log-polar vision selectively reduces the amount of data to be processed and simplifies several vision algorithms, making possible their implementation using few hardware resources. The reconfigurable architecture design has been devoted to implementation, and has been employed in an autonomous platform, which has power consumption, size and weight restrictions. Two different vision algorithms have been implemented in the reconfigurable pipeline, for which some experimental results are shown. Received: 30 March 2001 / Accepted: 11 February 2002 RID="*" ID="*" This work has been supported by the Ministerio de Ciencia y Tecnología and FEDER under project TIC2001-3546 Correspondence to: J.A. Boluda