Three-dimensional shape from color photometric stereo

Computer vision systems can be used to determine the shapes of real three-dimensional objects for purposes of object recognition and pose estimation or for CAD applications. One method that has been developed is photometric stereo. This method uses several images taken from the same viewpoint, but with different lightings, to determine the three-dimensional shape of an object. Most previous work in photometric stereo has been with gray-tone images; color images have only been used for dielectric materials. In this paper we describe a procedure for color photometric stereo, which recovers the shape of a colored object from two or more color images of the object under white illumination. This method can handle different types of materials, such as composites and metals, and can employ various reflection models such as the Lambertian, dichromatic, and Torrance-Sparrow models. For composite materials, colored metals, and dielectrics, there are two advantages of utilizing color information: at each pixel, there are more constraints on the orientation, and the result is less sensitive to noise. Consequently, the shape can be found more accurately. The method has been tested on both artificial and real images of objects of various materials, and on real images of a multi-colored object.     

Subspace position measurement in the presence of occlusion

An eigenspace-based method for measuring camera and/or object position in the presence of occlusions is described. Following the formation of a global eigenspace during a training phase, input images are divided into subsections for which a local set of projections is stored. Local occlusions in each subsection are detected by eigenspace reconstruction error. Only unoccluded subsections are then recombined dynamically to implement the nearest-neighbor search of the manifold and to avoid ambiguity in the subspace. Experimental results are described to demonstrate this technique for the measurement of the position of two metallic parts in the presence of significant occlusion.     

Occlusion filling in stereo: Theory and experiments

A number of stereo matching algorithms have been developed in the last few years, which also have successfully detected occlusions in stereo images. These algorithms typically fall short of a systematic study of occlusions; they predominantly emphasize matching and regard occlusion filling as a secondary operation. Filling occlusions, however, is useful in many applications such as image-based rendering where 3D models are desired to be as complete as possible. In this paper, we study occlusions in a systematic way and propose two algorithms to fill occlusions reliably by applying statistical modeling, visibility constraints, and scene constraints. We introduce a probabilistic, model-based filling order of the occluded points to maintain consistency in filling. Furthermore, we show how an ambiguity in the interpolation of the disparity value of an occluded point can safely be avoided using color homogeneity when the point’s neighborhood consists of multiple scene surfaces. We perform a comparative study and show that statistically, the new algorithms deliver good quality results compared to existing algorithms.     

3D Shape Recovery of Smooth Surfaces: Dropping the Fixed-Viewpoint Assumption

We present a new method for recovering the 3D shape of a featureless smooth surface from three or more calibrated images illuminated by different light sources (three of them are independent). This method is unique in its ability to handle images taken from unconstrained perspective viewpoints and unconstrained illumination directions. The correspondence between such images is hard to compute and no other known method can handle this problem locally from a small number of images. Our method combines geometric and photometric information in order to recover dense correspondence between the images and accurately computes the 3D shape. Only a single pass starting at one point and local computation are used. This is in contrast to methods that use the occluding contours recovered from many images to initialize and constrain an optimization process. The output of our method can be used to initialize such processes. In the special case of fixed viewpoint, the proposed method becomes a new perspective photometric stereo algorithm. Nevertheless, the introduction of the multiview setup, self-occlusions, and regions close to the occluding boundaries are better handled, and the method is more robust to noise than photometric stereo. Experimental results are presented for simulated and real images.     

Reconstructing a 3-D depth map from one or more images

Several algorithms are suggested for recovering depth and orientation maps of a surface from its image intensities. They combine the advantages of stereo vision and shape-from-shading (SFS) methods. These algorithms generate accurate, unambiguous and dense surface depth and orientation maps. Most of the existing SFS algorithms cannot be directly extended to combine stereo images because the recovery of surface depth and that of orientation are separated in these formulations. We first present an SFS algorithm that couples the generation of depth and orientation maps. This formulation also ensures that the reconstructed surface depth and its orientation are consistent. The SFS algorithm for a single image is then extended to utilize stereo images. The correspondence over stereo images is established simultaneously with the generation of surface depth and orientation. An alternative approach is also suggested for combining stereo and SFS techniques. This approach can be used to combine needle maps which are directly available from other sources such as photometric stereo. Finally we present an algorithm to combine sparse depth measurements with an orientation map to reconstruct a surface. The same algorithm can be combined with the above algorithms for solving the SFS problem with sparse depth measurements. Thus various information sources can be used to accurately reconstruct a surface.     

自适应成本量的抗遮挡光场深度估计算法

目的 光场相机通过一次成像同时记录场景的空间信息和角度信息,获取多视角图像和重聚焦图像,在深度估计中具有独特优势。遮挡是光场深度估计中的难点问题之一,现有方法没有考虑遮挡或仅仅考虑单一遮挡情况,对于多遮挡场景点,方法失效。针对遮挡问题,在多视角立体匹配框架下,提出了一种对遮挡鲁棒的光场深度估计算法。方法 首先利用数字重聚焦算法获取重聚焦图像,定义场景的遮挡类型,并构造相关性成本量。然后根据最小成本原则自适应选择最佳成本量,并求解局部深度图。最后利用马尔可夫随机场结合成本量和平滑约束,通过图割算法和加权中值滤波获取全局优化深度图,提升深度估计精度。结果 实验在HCI合成数据集和Stanford Lytro Illum实际场景数据集上展开,分别进行局部深度估计与全局深度估计实验。实验结果表明,相比其他先进方法,本文方法对遮挡场景效果更好,均方误差平均降低约26.8%。结论 本文方法能够有效处理不同遮挡情况,更好地保持深度图边缘信息,深度估计结果更准确,且时效性更好。此外,本文方法适用场景是朗伯平面场景,对于含有高光的非朗伯平面场景存在一定缺陷。     

Translational photometric alignment of single-view image sequences

Photometric stereo is a well-established method to estimate surface normals of an object. When coupled with depth-map estimation, it can be used to reconstruct an object’s height field. Typically, photometric stereo requires an image sequence of an object under the same viewpoint but with differing illumination directions. One crucial assumption of this configuration is perfect pixel correspondence across images in the sequence. While this assumption is often satisfied, certain setups are susceptible to translational errors or misalignments across images. Current methods to align image sequences were not designed specifically for single-view photometric stereo. Thus, they either struggle to account for changing illumination across images, require training sets, or are overly complex for these conditions. However, the unique nature of single-view photometric stereo allows one to model misaligned image sequences using the underlying image formation model and a set of translational shifts. This paper introduces such a technique, entitled translational photometric alignment, that employs the Lambertian model of image formation. This reduces the alignment problem to minimizing a nonlinear sum-squared error function in order to best reconcile the observed images with the generative model. Thus, the end goal of translational photometric alignment is not only to align image sequences, but also to produce the best surface-normal estimates given the observed images. Controlled experiments on the Yale Face Database B demonstrate the high accuracy of translational photometric alignment. The utility and benefits of the technique are further illustrated by additional experiments on image sequences suffering from uncontrolled real-world misalignments.     

Stereoscopic Segmentation

We cast the problem of multiframe stereo reconstruction of a smooth shape as the global region segmentation of a collection of images of the scene. Dually, the problem of segmenting multiple calibrated images of an object becomes that of estimating the solid shape that gives rise to such images. We assume that the radiance of the scene results in piecewise homogeneous image statistics. This simplifying assumption covers Lambertian scenes with constant albedo as well as fine homogeneous textures, which are known challenges to stereo algorithms based on local correspondence. We pose the segmentation problem within a variational framework, and use fast level set methods to find the optimal solution numerically. Our algorithm does not work in the presence of strong photometric features, where traditional reconstruction algorithms do. It enjoys significant robustness to noise under the assumptions it is designed for.     

The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows

We present an algorithm for separating the local gradient information and Lambertian color by using 4-source color photometric stereo in the presence of highlights and shadows. We assume that the surface reflectance can be approximated by the sum of a Lambertian and a specular component. The conventional photometric method is generalized for color images. Shadows and highlights in the input images are detected using either spectral or directional cues and excluded from the recovery process, thus giving more reliable estimates of local surface parameters.     

Robust active stereo vision using Kullback-Leibler divergence

Active stereo vision is a method of 3D surface scanning involving the projecting and capturing of a series of light patterns where depth is derived from correspondences between the observed and projected patterns. In contrast, passive stereo vision reveals depth through correspondences between textured images from two or more cameras. By employing a projector, active stereo vision systems find correspondences between two or more cameras, without ambiguity, independent of object texture. In this paper, we present a hybrid 3D reconstruction framework that supplements projected pattern correspondence matching with texture information. The proposed scheme consists of using projected pattern data to derive initial correspondences across cameras and then using texture data to eliminate ambiguities. Pattern modulation data are then used to estimate error models from which Kullback-Leibler divergence refinement is applied to reduce misregistration errors. Using only a small number of patterns, the presented approach reduces measurement errors versus traditional structured light and phase matching methodologies while being insensitive to gamma distortion, projector flickering, and secondary reflections. Experimental results demonstrate these advantages in terms of enhanced 3D reconstruction performance in the presence of noise, deterministic distortions, and conditions of texture and depth contrast.     

Combinatorial photometric stereo and its application in 3D modeling of melanoma

This article concerns a new type of photometric stereo algorithm for which outliers such as highlights and shadows, including attached and cast shadow, are mixed with Lambertian data. The underlying motivation behind this algorithm is very simple: an axial symmetrical setup in 6-light photometric stereo can be used to offer advantages. We investigate why an axial symmetrical setup is useful and how it can be used to improve standard photometric stereo. The main result is summarized as a combinatorial photometric stereo algorithm which embeds a non-Lambertian detection procedure. To apply this algorithm, it involves three steps. First, it combines a group of reflectance intensities to make five virtual images, whose equivalence is guaranteed due to the axial symmetrical setup of the 6-source photometric stereo system. Second, comparison between these virtual images generates a five by five skew-symmetric matrix. The Frobenius norm of this matrix is then employed as an index to determine whether there is a non-Lambertian pixel present among the six pixels. Finally, after identification of non-Lambertian pixels, standard photometric stereo is performed to realize 3D modeling. Validation of this algorithm has been conducted with both synthetic and real images. The real images were obtained from a newly designed 3D imaging device, the Skin Analyzer, for clinical inspection of melanoma. Experimental study shows that combinatorial photometric stereo gives promising results in suppressing shadows and highlights, while improving 3D reconstruction results. Furthermore, error analysis illustrates how to determine an appropriate threshold value to enable the algorithm to achieve optimal performance.     

基于分割稳健而快速的局部立体匹配及医学应用

为了快速消除双目立体匹配的歧义性,提出一种基于局部信息和分割、快速高效的两步立体匹配算法.首先分割彩色立体图像对,并基于改进的Geman-McClure函数得到初始匹配成本;在水平、垂直双方向上采用分割自适应权重方法,消除匹配特征的相似歧义,计算鲁棒匹配代价,择优选取初始视差.为了最优地分配遮挡等歧义区域,采用贪婪策略估计视差,包括不可靠视差检测、基于分割窄遮挡处理、基于极线最小二乘填充及基于映射的视差组合.实验结果表明,该算法结构简单、计算快速高效,能有效地消除匹配歧义,得到分段平滑、精度高的稠密视差图;可在临床医学疾病诊断应用中自动地为计算机辅助诊断系统提供可靠深度信息感知.     

LED-Based Photometric Stereo: Modeling,Calibration and Numerical Solution

We conduct a thorough study of photometric stereo under nearby point light source illumination, from modeling to numerical solution, through calibration. In the classical formulation of photometric stereo, the luminous fluxes are assumed to be directional, which is very difficult to achieve in practice. Rather, we use light-emitting diodes to illuminate the scene to be reconstructed. Such point light sources are very convenient to use, yet they yield a more complex photometric stereo model which is arduous to solve. We first derive in a physically sound manner this model, and show how to calibrate its parameters. Then, we discuss two state-of-the-art numerical solutions. The first one alternatingly estimates the albedo and the normals, and then integrates the normals into a depth map. It is shown empirically to be independent from the initialization, but convergence of this sequential approach is not established. The second one directly recovers the depth, by formulating photometric stereo as a system of nonlinear partial differential equations (PDEs), which are linearized using image ratios. Although the sequential approach is avoided, initialization matters a lot and convergence is not established either. Therefore, we introduce a provably convergent alternating reweighted least-squares scheme for solving the original system of nonlinear PDEs. Finally, we extend this study to the case of RGB images.     

The Geometry and Matching of Lines and Curves Over Multiple Views

This paper describes the geometry of imaged curves in two and three views. Multi-view relationships are developed for lines, conics and non-algebraic curves. The new relationships focus on determining the plane of the curve in a projective reconstruction, and in particular using the homography induced by this plane for transfer from one image to another. It is shown that given the fundamental matrix between two views, and images of the curve in each view, then the plane of a conic may be determined up to a two fold ambiguity, but local curvature of a curve uniquely determines the plane. It is then shown that given the trifocal tensor between three views, this plane defines a homography map which may be used to transfer a conic or the curvature from two views to a third. Simple expressions are developed for the plane and homography in each case.A set of algorithms are then described for automatically matching individual line segments and curves between images. The algorithms use both photometric information and the multiple view geometric relationships. For image pairs the homography facilitates the computation of a neighbourhood cross-correlation based matching score for putative line/curve correspondences. For image triplets cross-correlation matching scores are used in conjunction with line/curve transfer based on the trifocal geometry to disambiguate matches. Algorithms are developed for both short and wide baselines. The algorithms are robust to deficiencies in the segment extraction and partial occlusion.Experimental results are given for image pairs and triplets, for varying motions between views, and for different scene types. The methods are applicable to line/curve matching in stereo and trinocular rigs, and as a starting point for line/curve matching through monocular image sequences.     

An approximation technique for photometric stereo

Photometric stereo is a technique to determine the orientation of an object. It takes at least three images corresponding to three different light sources to determine the orientation and to eliminate the ambiguities which will arise if only two light sources are used. In this report we show that a plausible shape can be obtained based only on two light sources. The method is based on the approximation of a surface patch by a spherical patch. Two real images and one synthetic image are used to illustrate the method.     

Robust multiscale stereo matching from fundus images with radiometric differences

A robust multiscale stereo matching algorithm is proposed to find reliable correspondences between low contrast and weakly textured retinal image pairs with radiometric differences. Existing algorithms designed to deal with piecewise planar surfaces with distinct features and Lambertian reflectance do not apply in applications such as 3D reconstruction of medical images including stereo retinal images. In this paper, robust pixel feature vectors are formulated to extract discriminative features in the presence of noise in scale space, through which the response of low-frequency mechanisms alter and interact with the response of high-frequency mechanisms. The deep structures of the scene are represented with the evolution of disparity estimates in scale space, which distributes the matching ambiguity along the scale dimension to obtain globally coherent reconstructions. The performance is verified both qualitatively by face validity and quantitatively on our collection of stereo fundus image sets with ground truth, which have been made publicly available as an extension of standard test images for performance evaluation.     

Appearance characterization of linear Lambertian objects, generalized photometric stereo, and illumination-invariant face recognition

Traditional photometric stereo algorithms employ a Lambertian reflectance model with a varying albedo field and involve the appearance of only one object. In this paper, we generalize photometric stereo algorithms to handle all appearances of all objects in a class, in particular the human face class, by making use of the linear Lambertian property. A linear Lambertian object is one which is linearly spanned by a set of basis objects and has a Lambertian surface. The linear property leads to a rank constraint and, consequently, a factorization of an observation matrix that consists of exemplar images of different objects (e.g., faces of different subjects) under different, unknown illuminations. Integrability and symmetry constraints are used to fully recover the subspace bases using a novel linearized algorithm that takes the varying albedo field into account. The effectiveness of the linear Lambertian property is further investigated by using it for the problem of illumination-invariant face recognition using just one image. Attached shadows are incorporated in the model by a careful treatment of the inherent nonlinearity in Lambert's law. This enables us to extend our algorithm to perform face recognition in the presence of multiple illumination sources. Experimental results using standard data sets are presented     

Imposing Semi-Local Geometric Constraints for Accurate Correspondences Selection in Structure from Motion: A Game-Theoretic Perspective

Most Structure from Motion pipelines are based on the iterative refinement of an initial batch of feature correspondences. Typically this is performed by selecting a set of match candidates based on their photometric similarity; an initial estimate of camera intrinsic and extrinsic parameters is then computed by minimizing the reprojection error. Finally, outliers in the initial correspondences are filtered by enforcing some global geometric property such as the epipolar constraint. In the literature many different approaches have been proposed to deal with each of these three steps, but almost invariably they separate the first inlier selection step, which is based only on local image properties, from the enforcement of global geometric consistency. Unfortunately, these two steps are not independent since outliers can lead to inaccurate parameter estimation or even prevent convergence, leading to the well known sensitivity of all filtering approaches to the number of outliers, especially in the presence of structured noise, which can arise, for example, when the images present several repeated patterns. In this paper we introduce a novel stereo correspondence selection scheme that casts the problem into a Game-Theoretic framework in order to guide the inlier selection towards a consistent subset of correspondences. This is done by enforcing geometric constraints that do not depend on full knowledge of the motion parameters but rather on some semi-local property that can be estimated from the local appearance of the image features. The practical effectiveness of the proposed approach is confirmed by an extensive set of experiments and comparisons with state-of-the-art techniques.     

Noise Reduction in Surface Reconstruction from a Given Gradient Field

We present a gradient space technique for noise reduction in surfaces reconstructed from a noisy gradient field. We first analyze the error sources in the recovered gradient field of a surface using a three-image photometric stereo method. Based on this analysis, we propose an additive noise model to describe the errors in the surface gradient estimates. We then use a vector space formulation and construct a multiscale orthonormal expansion for gradient fields. Using the sparse representation properties of this expansion, we develop techniques for reducing the gradient field noise by coefficient selection with thresholding. The simulation results indicate that the proposed technique provides significant improvement on the noise levels of both the estimated gradient fields and the reconstructed surfaces under heavy noise levels. Furthermore, the experiments using noisy photometric stereo image triplets of real range data suggest that the additive model remains viable after the nonlinear photometric stereo operation to provide accurate noise removal.