眼球光心标定与距离修正的3维注视点估计

目的 在基于双目视线相交方法进行3维注视点估计的过程中,眼球光心3维坐标手工测量存在较大误差,且3维注视点估计结果在深度距离方向偏差较大。为此,提出了眼球光心标定与距离修正的方案对3维注视点估计模型进行改进。方法 首先,通过图像处理算法获取左右眼的PCCR（pupil center cornea reflection）矢量信息,并使用二阶多项式映射函数得到左、右眼的2维平面注视点;其次,通过眼球光心标定方法获取眼球光心的3维坐标,避免手工测量方法引入的误差;然后,结合平面注视点得到左、右眼的视线方向,计算视线交点得到初步的3维注视点;最后,针对结果在深度距离方向抖动较大的问题,使用深度方向数据滤波与Z平面截取修正法对3维注视点结果进行修正处理。结果 选择两个不同大小的空间测试,实验结果表明该方法在3050 cm的工作距离内,角度偏差0.7°,距离偏差17.8 mm,在50130 cm的工作距离内,角度偏差1.0°,距离偏差117.4 mm。与其他的3维注视点估计方法相比较,在同样的测试空间条件下,角度偏差和距离偏差均显著减小。结论 提出的眼球光心标定方法可以方便准确地获取眼球光心的3维坐标,避免手工测量方法带来的误差,对角度偏差的减小效果显著。提出的深度方向数据滤波与Z平面截取修正法可以有效抑制数据结果的抖动,对距离偏差的减小效果显著。     

Transfer of Fixation Using Affine Structure: Extending the Analysis to Stereo

This paper describes an algorithm for maintaining fixation upon a 3D body-centred point using 3D affine transfer, extending an earlier monocular method to stereo cameras. Transfer is based on corners detected in the image and matched over time and in stereo. The paper presents a method using all available matched data, providing immunity to noise and poor conditioning. The algorithm, implemented at video rates on a multi-processor machine, incorporates controlled degradation in the presence of insufficient data. Results are given from experiments using a four-axis active stereo camera platform, first which show the greater stability of the fixation point over the monocular method, both as it appears in the image and occurs in the scene; and, secondly, which show the recovery and evolution of 3D affine structure during fixation. It is shown that fixation and explicit structure recovery can occur separately, allowing the information required for gaze control to be computed in a fixed time.     

局部双目视差回归的目标距离估计

目的 双目视觉是目标距离估计问题的一个很好的解决方案。现有的双目目标距离估计方法存在估计精度较低或数据准备较繁琐的问题,为此需要一个可以兼顾精度和数据准备便利性的双目目标距离估计算法。方法 提出一个基于R-CNN（region convolutional neural network）结构的网络,该网络可以实现同时进行目标检测与目标距离估计。双目图像输入网络后,通过主干网络提取特征,通过双目候选框提取网络以同时得到左右图像中相同目标的包围框,将成对的目标框内的局部特征输入目标视差估计分支以估计目标的距离。为了同时得到左右图像中相同目标的包围框,使用双目候选框提取网络代替原有的候选框提取网络,并提出了双目包围框分支以同时进行双目包围框的回归;为了提升视差估计的精度,借鉴双目视差图估计网络的结构,提出了一个基于组相关和3维卷积的视差估计分支。结果 在KITTI（Karlsruhe Institute of Technology and Toyota Technological Institute）数据集上进行验证实验,与同类算法比较,本文算法平均相对误差值约为3.2%,远小于基于双目视差图估计算法（11.3%）,与基于3维目标检测的算法接近（约为3.9%）。另外,提出的视差估计分支改进对精度有明显的提升效果,平均相对误差值从5.1%下降到3.2%。通过在另外采集并标注的行人监控数据集上进行类似实验,实验结果平均相对误差值约为4.6%,表明本文方法可以有效应用于监控场景。结论 提出的双目目标距离估计网络结合了目标检测与双目视差估计的优势,具有较高的精度。该网络可以有效运用于车载相机及监控场景,并有希望运用于其他安装有双目相机的场景。     

Analysis of Error in Depth Perception with Vergence and Spatially Varying Sensing

In stereo vision the depth of a 3-D point is estimated based on the position of its projections on the left and right images. The image plane of cameras that produces the images consists of discrete pixels. This discretization of images generates uncertainty in estimation of the depth at each 3-D point. In this paper, we investigate the effect of vergence and spatially varying resolution on the depth estimation error. First, vergence is studied when pairs of stereo images with uniform resolution are used. Then the problem is studied for a stereo system similar to that of humans, in which cameras have high resolution in the center and nonlinearly decreasing resolution toward the periphery. In this paper we are only concerned with error in depth perception, assuming that stereo matching is already done.     

Camera calibration and three-dimensional world reconstruction of stereo-vision using neural networks

Stereo-pair images obtained from two cameras can be used to compute three-dimensional (3D) world coordinates of a point using triangulation. However, to apply this method, camera calibration parameters for each camera need to be experimentally obtained. Camera calibration is a rigorous experimental procedure in which typically 12 parameters are to be evaluated for each camera. The general camera model is often such that the system becomes nonlinear and requires good initial estimates to converge to a solution. We propose that, for stereo vision applications in which real-world coordinates are to be evaluated, artificial neural networks be used to train the system such that the need for camera calibration is eliminated. The training set for our neural network consists of a variety of stereo-pair images and corresponding 3D world coordinates. We present the results obtained on our prototype mobile robot that employs two cameras as its sole sensors and navigates through simple regular obstacles in a high-contrast environment. We observe that the percentage errors obtained from our set-up are comparable with those obtained through standard camera calibration techniques and that the system is accurate enough for most machine-vision applications.     

基于双目视觉的人脸三维重建

基于双目立体匹配算法PatchMatch算法,提出了一种获取人脸三维点云的算法。该算法对局部立体匹配算法PatchMatch进行了优化。该方法既不需要昂贵的设备,也不需要通用的人脸三维模型,而是结合了人脸的拓扑结构信息以及立体视觉局部优化算法。此方法采用非接触式的双目视觉采集技术获取左右视角的人脸图像,利用回归树集合(ensemble of regression trees,ERT)算法对人脸图像进行关键点定位,恢复人脸稀疏的视差估计,运用线性插值方法初步估计脸部的稠密视差值,并结合局部立体匹配算法对得到的视差结果进行平滑处理,重建人脸的三维点云信息。实验结果表明,这种算法能够还原出光滑的稠密人脸三维点云信息,在人脸Bosphorus数据库上取得了更加准确的人脸重建结果。     

Gaze direction estimation using support vector machine with active appearance model

In recent years, research on human-computer interaction is becoming popular, most of which uses body movements, gestures or eye gaze direction. Until now, gazing estimation is still an active research domain. We propose an efficient method to solve the problem of the eye gaze point. We first locate the eye region by modifying the characteristics of the Active Appearance Model (AAM). Then by employing the Support Vector Machine (SVM), we estimate the five gazing directions through classification. The original 68 facial feature points in AAM are modified into 36 eye feature points. According to the two-dimensional coordinates of feature points, we classify different directions of eye gazing. The modified 36 feature points describe the contour of eyes, iris size, iris location, and the position of pupils. In addition, the resolution of cameras does not affect our method to determine the direction of line of sight accurately. The final results show the independence of classifications, less classification errors, and more accurate estimation of the gazing directions.     

双目立体视觉的三维人脸重建方法

创建逼真的三维人脸模型始终是一个极具挑战性的课题.随着三维人脸模型在虚拟现实、视频监控、三维动画、人脸识别等领域的广泛应用,三维人脸重建成为计算机图像学和计算机视觉领域的一个研究热点.针对这一问题,提出一种基于双目立体视觉的三维人脸重建方法,重建过程中无需三维激光扫描仪和通用人脸模型.首先利用标定的2台摄像机获取人脸正面图像对,通过图像校正使图像对的极线对齐并且补偿摄像机镜头的畸变;在立体匹配方面,选择具有准确可靠视差的人脸边缘特征点作为种子像素,以种子像素的视差作为区域生长的视差,在外极线约束、单调性约束以及对应匹配的边缘特征点的约束下,进行水平扫描线上的区域生长,从而得到整个人脸区域的视差图,提高了对应点匹配的速度和准确度;最后,根据摄像机标定结果和立体匹配生成的视差图计算人脸空间散乱点的三维坐标,对人脸的三维点云进行三角剖分、网格细分和光顺处理.实验结果表明,该方法能够生成光滑、逼真的三维人脸模型,证明了该算法的有效性.     

基于立体视觉的人脸三维空间位置定位方法*

针对基于姿势的自然人机交互接口设计中头部定位问题,提出了一种快速的人脸三维空间位置定位算法。首先采用级联肤色分类器与类Haar特征分类器的人脸检测器对左右路输入的图像进行快速的人脸区域定位;然后利用仿射模型匹配局部收敛性好、速度快的特点实现了左右路图像的人脸区域对齐;最后通过立体视觉原理恢复人脸的三维空间坐标。实验结果证明,提出的人脸三维空间位置定位方法速度快、定位精度较高。     

Automatic facial feature extraction and 3D face modeling using two orthogonal views with application to 3D face recognition

We present a fully automated algorithm for facial feature extraction and 3D face modeling from a pair of orthogonal frontal and profile view images of a person's face taken by calibrated cameras. The algorithm starts by automatically extracting corresponding 2D landmark facial features from both view images, then compute their 3D coordinates. Further, we estimate the coordinates of the features that are hidden in the profile view based on the visible features extracted in the two orthogonal face images. The 3D coordinates of the selected feature points obtained from the images are used first to align, then to locally deform the corresponding facial vertices of the generic 3D model. Preliminary experiments to assess the applicability of the resulted models for face recognition show encouraging results.     

A real-time gaze position estimation method based on a 3-D eye model.

This paper proposes a new gaze-detection method based on a 3-D eye position and the gaze vector of the human eyeball. Seven new developments compared to previous works are presented. First, a method of using three camera systems, i.e., one wide-view camera and two narrow-view cameras, is proposed. The narrow-view cameras use autozooming, focusing, panning, and tilting procedures (based on the detected 3-D eye feature position) for gaze detection. This allows for natural head and eye movement by users. Second, in previous conventional gaze-detection research, one or multiple illuminators were used. These studies did not consider specular reflection (SR) problems, which were caused by the illuminators when working with users who wore glasses. To solve this problem, a method based on dual illuminators is proposed in this paper. Third, the proposed method does not require user-dependent calibration, so all procedures for detecting gaze position operate automatically without human intervention. Fourth, the intrinsic characteristics of the human eye, such as the disparity between the pupillary and the visual axes in order to obtain accurate gaze positions, are considered. Fifth, all the coordinates obtained by the left and right narrow-view cameras, as well as the wide-view camera coordinates and the monitor coordinates, are unified. This simplifies the complex 3-D converting calculation and allows for calculation of the 3-D feature position and gaze position on the monitor. Sixth, to upgrade eye-detection performance when using a wide-view camera, the adaptive-selection method is used. This involves an IR-LED on/off scheme, an AdaBoost classifier, and a principle component analysis method based on the number of SR elements. Finally, the proposed method uses an eigenvector matrix (instead of simply averaging six gaze vectors) in order to obtain a more accurate final gaze vector that can compensate for noise. Experimental results show that the root mean square error of gaze detection was about 0.627 cm on a 19-in monitor. The processing speed of the proposed method (used to obtain the gaze position on the monitor) was 32 ms (using a Pentium IV 1.8-GHz PC). It was possible to detect the user's gaze position at real-time speed.     

Evolutionary-based 3D reconstruction using an uncalibrated stereovision system: application of building a panoramic object view

In this paper, we propose an original evolutionary-based method for 3D panoramic reconstruction from an uncalibrated stereovision system (USS). The USS is composed of five cameras located on an arc of a circle around the object to be analyzed. The main originality of this work concerns the process of the calculation of the 3D information. Actually, with our method, 3D coordinates are directly obtained without any prior estimation of the fundamental matrix. The method operates in two steps. Firstly, points of interest are detected in pairs of images acquired by two consecutive cameras of the USS are matched. And secondly, using evolutionary algorithms, we jointly compute the transformed matrix between the two images and the respective depth of the points of interest. The accuracy of the proposed method is validated through a comparison with the depth values obtained using a traditional method. In order to perform 3D panoramic object reconstruction, the process is repeated for all the pairs of consecutive cameras. The 3D points thus obtained throughout the successive steps of the process which correspond to the different points of interest, are combined in order to obtain a set of 3D points all around the analyzed object.     

Self-calibration of spherical rectification for a PTZ-stereo system

Since PTZ (pan–tilt–zoom) camera is able to obtain multi-view-angle and multi-resolution information, PTZ-stereo system using two PTZ cameras has much higher capability and flexibility compared with traditional stereo system. In this paper, we propose a self-calibration framework to deal with the calibration of spherical rectification, which can be deemed as a kind of relative pose estimation, for a PTZ-stereo system. The goal of this calibration is to guarantee high performance of stereo rectification, so that stereo matching can be achieved more efficiently and accurately. In this framework, we assume two PTZ cameras are fully calibrated, i.e., the focal length and the local camera orientation can be computed by given pan–tilt–zoom values. This approach, which is based on point matches, aims at finding uniformly distributed point matches in an iterative way. At each iteration, according to the distribution of previously used point matches, the system could automatically guide two cameras to move to collect a new match. Point matching is firstly performed for the lowest zoom setting (widest field of view). Once a candidate match is chosen, each camera is then controlled to zoom in on corresponding point to get a refined match with high spatial resolution. The final match will be added into the estimation to update the calibration parameters. Compared with previous researches, the proposed framework has the following advantages: (1) Neither manual interaction nor calibration object is needed. Calibration samples (point matches) will be added and removed in each stage automatically. (2) The distribution of calibration samples is as uniform as possible so that biased estimation could be avoided to some extent. (3) The accuracy of calibration can be controlled and improved when iteration goes on. These advantages make the proposed framework more practicable in applications. Experimental results illustrate its accuracy.     

基于立体视觉的平面圆参数高精度测量算法*

针对目前平面圆的圆心和半径测量方法存在的不足,提出一种基于立体视觉的平面圆参数高精度测量方法。该方法首先根据边缘分组算法和二次曲线椭圆拟合判别算法提取双目图像中的椭圆特征;其次利用分段立体匹配算法获取平面圆特征上部分点的三维坐标;最后通过三维平面拟合、坐标转换、平面圆拟合以及坐标反变换获取平面圆的圆心和半径参数。实验结果表明,该方法有效地减小了平面圆透视投影引起的畸变误差,提高了平面圆参数的测量精度。     

基于双相机捕获面部表情及人体姿态生成三维虚拟人动画

为了生成表情丰富、动作流畅的三维虚拟人动画,提出了一种基于双相机同步捕获面部表情及人体姿态生成三维虚拟人动画的方法。首先,采用传输控制协议（TCP）网络时间戳方法实现双相机时间同步,采用张正友标定法实现双相机空间同步。然后,利用双相机分别采集面部表情和人体姿态。采集面部表情时,提取图像的2D特征点,利用这些2D特征点回归计算得到面部行为编码系统（FACS）面部行为单元,为实现表情动画做准备;以标准头部3D坐标值为基准,根据相机内参,采用高效n点投影（EPnP）算法实现头部姿态估计;之后将面部表情信息和头部姿态估计信息进行匹配。采集人体姿态时,利用遮挡鲁棒姿势图（ORPM）方法计算人体姿态,输出每个骨骼点位置、旋转角度等数据。最后,在虚幻引擎4（UE4）中使用建立的虚拟人体三维模型来展示数据驱动动画的效果。实验结果表明,该方法能够同步捕获面部表情及人体姿态,而且在实验测试中的帧率达到20 fps,能实时生成自然真实的三维动画。     

双足机器人实时障碍检测视觉系统

提出一种双足步行机器人的实时障碍检测视觉系统．基于图像平面与机器人行走地面之间的映射变换矩阵的唯一性准则，判别图像中的像点是否位于地面上，高于或低于地面的点被认为是障碍点，为减少实际行走过程中系统外部和内部参数变化对映射变换矩阵的影响，系统加入了在线校正映射变换矩阵模块．在提取出障碍物体边缘后，通过对其三维信息的简单恢复，建立了机器人行走空间的障碍投影图．该系统计算量小，可靠性强，能基本满足双足步行机器人实时避障的要求。     

A multimodal approach for 3D face modeling and recognition using 3D deformable facial mask

We present a multimodal approach for face modeling and recognition. The algorithm uses three cameras to capture stereo images, two frontal and one profile, of the face. 2D facial features are extracted from one of the frontal images and a dense disparity map is computed from the two frontal images. Using the extracted 2D features and their corresponding disparities, we compute their 3D coordinates. We next align a low resolution 3D mesh model to the 3D features, re-project its vertices onto the frontal 2D image and adjust its profile silhouette vertices using the profile view image. We increase the resolution of the resulting 2D model at its center region to obtain a facial mask model covering distinctive features of the face. The 2D coordinates of the vertices, along with their disparities, result in a deformed 3D mask model specific to a given subject’s face. Our method integrates information from the extracted facial features from the 2D image modality with information from the 3D modality obtained from the stereo images. Application of the models in 3D face recognition, for 112 subjects, validates the algorithm with a 95% identification rate and 92% verification rate at 0.1% false acceptance rate.

多介质下空间目标的视觉测量

基于计算机视觉的无接触三维测量原理和光的折射定律,提出一种多介质下空间目标视觉测量方法。采用多相机捆绑调整及多角度相互校正,通过双目立体视觉技术结合光线在多介质中发生两次折射的数学模型对运动体上各标志点进行测量,获取各标志点的三维空间坐标,经平面拟合、坐标转换,最终解算得到运动体六自由度空间坐标。实验结果表明,该方法适用于多介质视觉测量,具有较高的测量精度和良好的稳定性。     

Stereo with Oblique Cameras

Mosaics acquired by pushbroom cameras, stereo panoramas, omnivergent mosaics, and spherical mosaics can be viewed as images taken by non-central cameras, i.e. cameras that project along rays that do not all intersect at one point. It has been shown that in order to reduce the correspondence search in mosaics to a one-parametric search along curves, the rays of the non-central cameras have to lie in double ruled epipolar surfaces. In this work, we introduce the oblique stereo geometry, which has non-intersecting double ruled epipolar surfaces. We analyze the configurations of mutually oblique rays that see every point in space. These configurations, called oblique cameras, are the most non-central cameras among all cameras. We formulate the assumption under which two oblique cameras posses oblique stereo geometry and show that the epipolar surfaces are non-intersecting double ruled hyperboloids and two lines. We show that oblique cameras, and the correspondingoblique stereo geometry, exist and give an example of a physically realizable oblique stereo geometry. We introduce linear oblique cameras as those which can be generated by a linear mapping from points in space to camera rays and characterize those collineations which generate them. We show that all linear oblique cameras are obtained by a collineation from one example of an oblique camera. Finally, we relate oblique cameras to spreads known from incidence geometries.