Analytic PCA construction for theoretical analysis of lighting variability in images of a Lambertian object

We analyze theoretically the subspace best approximating images of a convex Lambertian object taken from the same viewpoint, but under different distant illumination conditions. We analytically construct the principal component analysis for images of a convex Lambertian object, explicitly taking attached shadows into account, and find the principal eigenmodes and eigenvalues with respect to lighting variability. Our analysis makes use of an analytic formula for the irradiance in terms of spherical-harmonic coefficients of the illumination and shows, under appropriate assumptions, that the principal components or eigenvectors are identical to the spherical harmonic basis functions evaluated at the surface normal vectors. Our main contribution is in extending these results to the single-viewpoint case, showing how the principal eigenmodes and eigenvalues are affected when only a limited subset (the upper hemisphere) of normals is available and the spherical harmonics are no longer orthonormal over the restricted domain. Our results are very close, both qualitatively and quantitatively, to previous empirical observations and represent the first essentially complete theoretical explanation of these observations.     

Harmonic Functions for Rotational Symmetry Vector Fields

Representing rotational symmetry vector as a set of vectors is not suitable for design due to lacking of a consistent ordering for measurement. In this paper we introduce a spectral method to find rotation invariant harmonic functions for symmetry vector field design. This method is developed for 3D vector fields, but it is applicable in 2D. Given the finite symmetry group G of a symmetry vector field v (x) on a 3D domain Ω, we formulate the harmonic function h(s) as a stationary point of group G. Using the real spherical harmonic (SH) bases, we showed the coefficients of the harmonic functions are an eigenvector of the SH rotation matrices corresponding to group G. Instead of solving eigen problems to obtain the eigenvector, we developed a forward constructive method based on orthogonal group theory. The harmonic function found by our method is not only invariant under G, but also expressive and can distinguish different rotations with respect to G. At last, we demonstrate some vector field design results with tetrahedron‐symmetry, cube‐symmetry and dodecahedron‐symmetry groups.     

A Quick Rendering Method Using Basis Functions for Interactive Lighting Design

When designing interior lighting effects, it is desirable to compare a variety of lighting designs involving different lighting devices and directions of light. It is, however, time-consuming to generate images with many different lighting parameters, taking interreflection into account, because all luminances must be calculated and recalculated. This makes it difficult to design lighting effects interactively. To address this problem, this paper proposes a method of quickly generating images of a given scene illustrating an interreflective environment illuminated by sources with arbitrary luminous intensity distributions. In the proposed method, the luminous intensity ditribution is expressed with basis functions. The proposed method uses a series of spherical harmonic functions as basis functions, and calculates in advance each intensity on surfaces lit by the light sources whose luminous intensity distribution are the same as the spherical harmonic functions. The proposed method makes it possible to generate images so quickly that we can change the luminous intensity distribution interactively. Combining the proposed method with an interactive walk-through that employs intensity mapping, an interactive system for lighting design is implemented. The usefulness of the proposed method is demonstrated by its application to interactive lighting design, where many images are generated by altering lighting devices and/or direction of light.     

Lambertian reflectance and linear subspaces

We prove that the set of all Lambertian reflectance functions (the mapping from surface normals to intensities) obtained with arbitrary distant light sources lies close to a 9D linear subspace. This implies that, in general, the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace, explaining prior empirical results. We also provide a simple analytic characterization of this linear space. We obtain these results by representing lighting using spherical harmonics and describing the effects of Lambertian materials as the analog of a convolution. These results allow us to construct algorithms for object recognition based on linear methods as well as algorithms that use convex optimization to enforce nonnegative lighting functions. We also show a simple way to enforce nonnegative lighting when the images of an object lie near a 4D linear space. We apply these algorithms to perform face recognition by finding the 3D model that best matches a 2D query image.     

Face Relighting from a Single Image under Arbitrary Unknown Lighting Conditions

In this paper, we present a new method to modify the appearance of a face image by manipulating the illumination condition, when the face geometry and albedo information is unknown. This problem is particularly difficult when there is only a single image of the subject available. Recent research demonstrates that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace using a spherical harmonic representation. Moreover, morphable models are statistical ensembles of facial properties such as shape and texture. In this paper, we integrate spherical harmonics into the morphable model framework by proposing a 3D spherical harmonic basis morphable model (SHBMM). The proposed method can represent a face under arbitrary unknown lighting and pose simply by three low-dimensional vectors, i.e., shape parameters, spherical harmonic basis parameters, and illumination coefficients, which are called the SHBMM parameters. However, when the image was taken under an extreme lighting condition, the approximation error can be large, thus making it difficult to recover albedo information. In order to address this problem, we propose a subregion-based framework that uses a Markov random field to model the statistical distribution and spatial coherence of face texture, which makes our approach not only robust to extreme lighting conditions, but also insensitive to partial occlusions. The performance of our framework is demonstrated through various experimental results, including the improved rates for face recognition under extreme lighting conditions.     

Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics

In this paper, we propose two novel methods for face recognition under arbitrary unknown lighting by using spherical harmonics illumination representation, which require only one training image per subject and no 3D shape information. Our methods are based on the result which demonstrated that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace. We provide two methods to estimate the spherical harmonic basis images spanning this space from just one image. Our first method builds the statistical model based on a collection of 2D basis images. We demonstrate that, by using the learned statistics, we can estimate the spherical harmonic basis images from just one image taken under arbitrary illumination conditions if there is no pose variation. Compared to the first method, the second method builds the statistical models directly in 3D spaces by combining the spherical harmonic illumination representation and a 3D morphable model of human faces to recover basis images from images across both poses and illuminations. After estimating the basis images, we use the same recognition scheme for both methods: we recognize the face for which there exists a weighted combination of basis images that is the closest to the test face image. We provide a series of experiments that achieve high recognition rates, under a wide range of illumination conditions, including multiple sources of illumination. Our methods achieve comparable levels of accuracy with methods that have much more onerous training data requirements. Comparison of the two methods is also provided.     

Analytic Tangent Irradiance Environment Maps for Anisotropic Surfaces

Environment‐mapped rendering of Lambertian isotropic surfaces is common, and a popular technique is to use a quadratic spherical harmonic expansion. This compact irradiance map representation is widely adopted in interactive applications like video games. However, many materials are anisotropic, and shading is determined by the local tangent direction, rather than the surface normal. Even for visualization and illustration, it is increasingly common to define a tangent vector field, and use anisotropic shading. In this paper, we extend spherical harmonic irradiance maps to anisotropic surfaces, replacing Lambertian reflectance with the diffuse term of the popular Kajiya‐Kay model. We show that there is a direct analogy, with the surface normal replaced by the tangent. Our main contribution is an analytic formula for the diffuse Kajiya‐Kay BRDF in terms of spherical harmonics; this derivation is more complicated than for the standard diffuse lobe. We show that the terms decay even more rapidly than for Lambertian reflectance, going as l^–3, where l is the spherical harmonic order, and with only 6 terms (l = 0 and l = 2) capturing 99.8% of the energy. Existing code for irradiance environment maps can be trivially adapted for real‐time rendering with tangent irradiance maps. We also demonstrate an application to offline rendering of the diffuse component of fibers, using our formula as a control variate for Monte Carlo sampling.     

Guaranteed estimation of signals with bounded variances of derivatives

Consideration is given to the problem of signal estimation against the background of the white noise when the information about the signal is represented in the form of numerical characteristics such as constraints on the variance of the signal itself and variances of some its derivatives. We proposed a method to solve this problem using the technique of filtering in the time domain by minimizing the functional that is a combination of the H ₂-norm of the transfer function from the measurement noise to the error of estimation and the H _∞-norm of the transfer function from the generating noise to the error of estimation.     

Efficient visibility encoding for dynamic illumination in direct volume rendering

We present an algorithm that enables real-time dynamic shading in direct volume rendering using general lighting, including directional lights, point lights, and environment maps. Real-time performance is achieved by encoding local and global volumetric visibility using spherical harmonic (SH) basis functions stored in an efficient multiresolution grid over the extent of the volume. Our method enables high-frequency shadows in the spatial domain, but is limited to a low-frequency approximation of visibility and illumination in the angular domain. In a first pass, level of detail (LOD) selection in the grid is based on the current transfer function setting. This enables rapid online computation and SH projection of the local spherical distribution of visibility information. Using a piecewise integration of the SH coefficients over the local regions, the global visibility within the volume is then computed. By representing the light sources using their SH projections, the integral over lighting, visibility, and isotropic phase functions can be efficiently computed during rendering. The utility of our method is demonstrated in several examples showing the generality and interactive performance of the approach.     

Neural Structures Using the Eigenstates of a Quantum Harmonic Oscillator

The main result of the paper is the use of orthogonal Hermite polynomials as the basis functions of feedforward neural networks. The proposed neural networks have some interesting properties: (i) the basis functions are invariant under the Fourier transform, subject only to a change of scale, (ii) the basis functions are the eigenstates of the quantum harmonic oscillator, and stem from the solution of Schrödinger's diffusion equation. The proposed feed-forward neural networks demonstrate the particle-wave nature of information and can be used in nonparametric estimation. Possible applications of the proposed neural networks include function approximation, image processing and system modelling.     

A heuristic search-based motion correspondence algorithm using fuzzy clustering

Motion correspondence problem between many feature points of consecutive frames is computationally explosive. We present a heuristic algorithm for finding out the most probable motion correspondence of points in consecutive frames, based on fuzzy confidence degrees. The proposed algorithm consists of three stages: (i) reduction of the search space for candidate points of association, (ii) pairwise association cost estimation and (iii) complete association of every feature point between the consecutive frames. In the first stage, all the points in a frame, frame t-1 are grouped into several groups by using fuzzy clustering. This is done with a Euclidean distance as a similarity measure between the points. The points in the following frame, frame t are also clustered into the same number of groups with respect to the cluster centers of the previous frame. The association between the points of the consecutive frames is allowed only for the points that belong to the same group in each frame. In the second stage, the cost of each association of a point in frame t-1 with a point in frame t is estimated by using motion constraints that are based on the velocity vector and the orientation angle of each point. The cost is measured as a fuzzy confidence degree of each head point, i.e., a point in frame t-1, belonging to each measurement, i.e., a point in frame t. In the final stage, we search for the most likely associations among all the possible mappings between the feature points in the consecutive frames. A search tree is constructed in such a way that an ith level node represents an association of ith node in frame t-1 with a node in frame t. We devise a heuristic function of an admissible A* algorithm by using the pairwise association cost developed in the second stage. Experimental results show an accuracy of more than 98%.     

Video frame interpolation via optical flow estimation with image inpainting

As we all know, video frame rate determines the quality of the video. The higher the frame rate, the smoother the movements in the picture, the clearer the information expressed, and the better the viewing experience for people. Video interpolation aims to increase the video frame rate by generating a new frame image using the relevant information between two consecutive frames, which is essential in the field of computer vision. The traditional motion compensation interpolation method will cause holes and overlaps in the reconstructed frame, and is easily affected by the quality of optical flow. Therefore, this paper proposes a video frame interpolation method via optical flow estimation with image inpainting. First, the optical flow between the input frames is estimated via combined local and global-total variation (CLG-TV) optical flow estimation model. Then, the intermediate frames are synthesized under the guidance of the optical flow. Finally, the nonlocal self-similarity between the video frames is used to solve the optimization problem, to fix the pixel loss area in the interpolated frame. Quantitative and qualitative experimental results show that this method can effectively improve the quality of optical flow estimation, generate realistic and smooth video frames, and effectively increase the video frame rate.     

All-Frequency Lighting with Multiscale Spherical Radial Basis Functions

This paper proposes a novel multiscale spherical radial basis function (MSRBF) representation for all-frequency lighting. It supports the illumination of distant environment as well as the local illumination commonly used in practical applications, such as games. The key is to define a multiscale and hierarchical structure of spherical radial basis functions (SRBFs) with basis functions uniformly distributed over the sphere. The basis functions are divided into multiple levels according to their coverage (widths). Within the same level, SRBFs have the same width. Larger width SRBFs are responsible for lower frequency lighting while the smaller width ones are responsible for the higher frequency lighting. Hence, our approach can achieve the true all-frequency lighting that is not achievable by the single-scale SRBF approach. Besides, the MSRBF approach is scalable as coarser rendering quality can be achieved without reestimating the coefficients from the raw data. With the homogeneous form of basis functions, the rendering is highly efficient. The practicability of the proposed method is demonstrated with real-time rendering and effective compression for tractable storage.     

Noise-resistant fitting for spherical harmonics

Spherical harmonic (SH) basis functions have been widely used for representing spherical functions in modeling various illumination properties. They can compactly represent low-frequency spherical functions. However, when the unconstrained least square method is used for estimating the SH coefficients of a hemispherical function, the magnitude of these SH coefficients could be very large. Hence, the rendering result is very sensitive to quantization noise (introduced by modern texture compression like S3TC, IEEE half float data type on GPU, or other lossy compression methods) in these SH coefficients. Our experiments show that, as the precision of SH coefficients are reduced, the rendered images may exhibit annoying visual artifacts. To reduce the noise sensitivity of the SH coefficients, this paper first discusses how the magnitude of SH coefficients affects the rendering result when there is quantization noise. Then, two fast fitting methods for estimating the noise-resistant SH coefficients are proposed. They can effectively control the magnitude of the estimated SH coefficients and, hence, suppress the rendering artifacts. Both statistical and visual results confirm our theory.     

深度时空能量特征表示下的人体行为识别

目的 利用深度图序列进行人体行为识别是机器视觉和人工智能中的一个重要研究领域,现有研究中存在深度图序列冗余信息过多以及生成的特征图中时序信息缺失等问题。针对深度图序列中冗余信息过多的问题,提出一种关键帧算法,该算法提高了人体行为识别算法的运算效率;针对时序信息缺失的问题,提出了一种新的深度图序列特征表示方法,即深度时空能量图（depth spatial-temporal energy map,DSTEM）,该算法突出了人体行为特征的时序性。方法 关键帧算法根据差分图像序列的冗余系数剔除深度图序列的冗余帧,得到足以表述人体行为的关键帧序列。DSTEM算法根据人体外形及运动特点建立能量场,获得人体能量信息,再将能量信息投影到3个正交轴获得DSTEM。结果 在MSR_Action3D数据集上的实验结果表明,关键帧算法减少冗余量,各算法在关键帧算法处理后运算效率提高了20% 30%。对DSTEM提取的方向梯度直方图（histogram of oriented gradient,HOG）特征,不仅在只有正序行为的数据库上识别准确率达到95.54%,而且在同时具有正序和反序行为的数据库上也能保持82.14%的识别准确率。结论 关键帧算法减少了深度图序列中的冗余信息,提高了特征图提取速率;DSTEM不仅保留了经过能量场突出的人体行为的空间信息,而且完整地记录了人体行为的时序信息,在带有时序信息的行为数据上依然保持较高的识别准确率。     

A Minimal Closed-form Solution for the Perspective Three Orthogonal Angles (P3oA) Problem: Application To Visual Odometry

We provide a simple closed-form solution to the Perspective three orthogonal angles (P3oA) problem: given the projection of three orthogonal lines in a calibrated camera, find their 3D directions. Upon this solution, an algorithm for the estimation of the camera relative rotation between two frames is proposed. The key idea is to detect triplets of orthogonal lines in a hypothesize-and-test framework and use all of them to compute the camera rotation in a robust way. This approach is suitable for human-made environments where numerous groups of orthogonal lines exist. We evaluate the numerical stability of the P3oA solution and the estimation of the relative rotation with synthetic and real data, comparing our results to other state-of-the-art approaches.     

Efficient Relighting of RBF-Based Illumination Adjustable Images

An illumination adjustable image (IAI) contains a large number of prerecorded images under various light directions. Relighting a scene under complicated lighting conditions can be achieved from the IAI. Using the radial basis function (RBF) approach to represent an IAI is proven to be more efficient than using the spherical harmonic approach. However, to represent high-frequency lighting effects, we need to use many RBFs. Hence, the relighting speed could be very slow. This brief investigates a partial reconstruction scheme for relighting an IAI based on the locality of RBFs. Compared with the conventional RBF and spherical harmonics (SH) approaches, the proposed scheme has a much faster relighting speed under the similar distortion performance.