Fast Shadow Removal Using Adaptive Multi‐Scale Illumination Transfer

In this paper, we present a new method for removing shadows from images. First, shadows are detected by interactive brushing assisted with a Gaussian Mixture Model. Secondly, the detected shadows are removed using an adaptive illumination transfer approach that accounts for the reflectance variation of the image texture. The contrast and noise levels of the result are then improved with a multi‐scale illumination transfer technique. Finally, any visible shadow boundaries in the image can be eliminated based on our Bayesian framework. We also extend our method to video data and achieve temporally consistent shadow‐free results.     

Augmented reality with automatic illumination control incorporatingellipsoidal models

In applications of augmented reality like virtual studio TV production, multisite video conference applications using a virtual meeting room and synthetic/natural hybrid coding according to the new ISO/MPEG-4 standard, a synthetic scene is mixed into a natural scene to generate a synthetic/natural hybrid image sequence. For realism, the illumination in both scenes should be identical. In this paper, the illumination of the natural scene is estimated automatically and applied to the synthetic scene. The natural scenes are restricted to scenes with nonoccluding, simple, moving, mainly rigid objects. For illumination estimation, these natural objects are automatically segmented in the natural image sequence and three-dimensionally (3-D) modeled using ellipsoid-like models. The 3-D shape, 3-D motion, and the displaced frame difference between two succeeding images are evaluated to estimate three illumination parameters. The parameters describe a distant point light source and ambient light. Using the estimated illumination parameters, the synthetic scene is rendered and mixed to the natural image sequence. Experimental results with a moving virtual object mixed into real video telephone sequences show that the virtual object appears naturally having the same shading and shadows as the real objects. Further, shading and shadow allows the viewer to understand the motion trajectory of the objects much better     

Moving shadow detection and removal for traffic sequences

Segmentation of moving objects in a video sequence is a basic task for application of computer vision. However, shadows extracted along with the objects can result in large errors in object localization and recognition. In this paper, we propose a method of moving shadow detection based on edge information, which can effectively detect the cast shadow of a moving vehicle in a traffic scene. Having confirmed shadows existing in a figure, we execute the shadow removal algorithm proposed in this paper to segment the shadow from the foreground. The shadow eliminating algorithm removes the boundary of the cast shadow and preserves object edges firstly; secondly, it reconstructs coarse object shapes based on the edge information of objects; and finally, it extracts the cast shadow by subtracting the moving object from the change detection mask and performs further processing. The proposed method has been further tested on images taken under different shadow orientations, vehicle colors and vehicle sizes, and the results have revealed that shadows can be successfully eliminated and thus good video segmentation can be obtained.     

An online spatio-temporal tensor learning model for visual tracking and its applications to facial expression recognition

Robust visual tracking remains a technical challenge in real-world applications, as an object may involve many appearance variations. In existing tracking frameworks, objects in an image are often represented as vector observations, which discounts the 2-D intrinsic structure of the image. By considering an image in its actual form as a matrix, we construct the 3rd order tensor based object representation to preserve the spatial correlation within the 2-D image and fully exploit the useful temporal information. We perform incremental update of the object template using the N-mode SVD to model the appearance variations, which reduces the influence of template drifting and object occlusions. The proposed scheme efficiently learns a low-dimensional tensor representation through adaptively updating the eigenbasis of the tensor. Tensor based Bayesian inference in the particle filter framework is then utilized to realize tracking. We present the validation of the proposed tracking system by conducting the real-time facial expression recognition with video data and a live camera. Experiment evaluation on challenging benchmark image sequences undergoing appearance variations demonstrates the significance and effectiveness of the proposed algorithm.     

Lighting Simulation of Augmented Outdoor Scene Based on a Legacy Photograph

We propose a novel approach to simulate the illumination of augmented outdoor scene based on a legacy photograph. Unlike previous works which only take surface radiosity or lighting related prior information as the basis of illumination estimation, our method integrates both of these two items. By adopting spherical harmonics, we deduce a linear model with only six illumination parameters. The illumination of an outdoor scene is finally calculated by solving a linear least square problem with the color constraint of the sunlight and the skylight. A high quality environment map is then set up, leading to realistic rendering results. We also explore the problem of shadow casting between real and virtual objects without knowing the geometry of objects which cast shadows. An efficient method is proposed to project complex shadows (such as tree's shadows) on the ground of the real scene to the surface of the virtual object with texture mapping. Finally, we present an unified scheme for image composition of a real outdoor scene with virtual objects ensuring their illumination consistency and shadow consistency. Experiments demonstrate the effectiveness and flexibility of our method.     

Illumination invariant object tracking with adaptive sparse representation

Since the introduction of the sparse representation-based tracking method named ?₁ tracker, there have been further studies into this tracking framework with promised results in challenging video sequences. However, in the situation of large illumination changes and shadow casting, the tracked object cannot be modeled efficiently by sparse representation templates. To overcome this problem, we propose a new illumination invariant tracker based on photometric normalization techniques and the sparse representation framework. With photometric normalization methods, we designed a new illumination invariant template presentation for tracking that eliminates the illumination influences, such as brightness variation and shadow casting. For a higher tracking accuracy, we introduced a strategy that adaptively selects the optimum template presentation at the update step of the tracking process. The experiments show that our approach outperforms the previous ?₁ and some state-of-the-art algorithms in tracking sequences with severe illumination effects.     

交通视频序列阴影检测算法研究

针对交通视频序列中的阴影检测问题,在分析目前常用的几种阴影检测算法的基础上,提出了一种基于交通视频流的灰度图阴影检测算法。该算法首先通过建立一个具有鲁棒性和自适应性的背景模型来获取背景,同时对阴影进行初步模糊滤除,然后再通过车体的重构来准确去除阴影。实践表明,该算法准确度高,具有广泛的应用前景。     

Online learning and fusion of orientation appearance models for robust rigid object tracking

We introduce a robust framework for learning and fusing of orientation appearance models based on both texture and depth information for rigid object tracking. Our framework fuses data obtained from a standard visual camera and dense depth maps obtained by low-cost consumer depth cameras such as the Kinect. To combine these two completely different modalities, we propose to use features that do not depend on the data representation: angles. More specifically, our framework combines image gradient orientations as extracted from intensity images with the directions of surface normals computed from dense depth fields. We propose to capture the correlations between the obtained orientation appearance models using a fusion approach motivated by the original Active Appearance Models (AAMs). To incorporate these features in a learning framework, we use a robust kernel based on the Euler representation of angles which does not require off-line training, and can be efficiently implemented online. The robustness of learning from orientation appearance models is presented both theoretically and experimentally in this work. This kernel enables us to cope with gross measurement errors, missing data as well as other typical problems such as illumination changes and occlusions. By combining the proposed models with a particle filter, the proposed framework was used for performing 2D plus 3D rigid object tracking, achieving robust performance in very difficult tracking scenarios including extreme pose variations.     

基于阴影检测的HSV空间自适应背景模型的车辆追踪检测

在交通监控中，要进行车辆的检测、车流量统计、实时追踪、车速测定等工作，而如何从复杂的背景中分割运动物体是至关重要的一步，目前采用的典型方法是背景相减方法。为了对运动车辆进行准确快速的检测，在研究了目前存在的各种方法之后，提出了一种新的基于阴影检测的HSV空间自适应背景模型的车辆追踪检测算法，并将其应用于运动物体的分割，同时给出了具体的试验结果。该方法之所以不在传统的RGB空间实现，而在HSV空间实现，因为HSV空间可以提供更丰富的颜色信息。运行试验结果表明，该方法准确率高，适应性强，运算速度快，兼具灵活性，能满足实时检测的需要。     

Moving Cast Shadows Detection Using Ratio Edge

Moving objects segmentation plays a very important role in real-time image analysis. However, as one of the common parts in the natural scenes, shadows severely interfere with the accuracy of moving objects detection in video surveillance. In this paper, we present a novel method for moving cast shadows detection. Based on the analysis of the physical model of moving shadows, we prove that the ratio edge is illumination invariant. The distribution of the ratio edge is discussed and a significance test is performed to classify each moving pixel into foreground object or moving shadow. Intensity constraint and geometric heuristics are imposed to further improve the performance. Experiments on various typical scenes exhibit the robustness of the proposed method. Extensively quantitative evaluation and comparison demonstrate that the proposed method significantly outperforms state-of-the-art methods.     

Shadow elimination and vehicles classification approaches in traffic video surveillance context

Video surveillance on highway is a hot topic and a great challenge in Intelligent Transportation Systems. In such applications requiring objects extraction, cast shadows induce shape distortions and object fusions interfering performance of high level algorithms. Shadow elimination allows to improve the performances of video object extraction, classification and tracking. In other hand, it is very important to recognize the type of a detected object in order to track reliably and estimate traffic parameters correctly. This paper presents two approaches to enhance automatic traffic surveillance systems. The first deals with the elimination of shadows and the second concerns the classification of vehicles, based on robust vision and image processing. For moving shadow elimination, a contrast model is proposed to describe and remove dynamic shadows based on the idea that a shadow transformation is a change in contrast. For vehicles classification, Hu moments are calculated in a manner to reduce the perspective effects and used to describe vehicles in knowledge base. Experimental results on the various challenging video sequences show that the proposed approach outperforms classification methods of related works (with a classification accuracy of 96.96%), and that the shadow elimination approach performs better than compared works (with detection rate of 95–99% and discrimination rate of 85.7–89%).     

A dynamic conditional random field model for foreground and shadow segmentation

This paper proposes a dynamic conditional random field (DCRF) model for foreground object and moving shadow segmentation in indoor video scenes. Given an image sequence, temporal dependencies of consecutive segmentation fields and spatial dependencies within each segmentation field are unified by a dynamic probabilistic framework based on the conditional random field (CRF). An efficient approximate filtering algorithm is derived for the DCRF model to recursively estimate the segmentation field from the history of observed images. The foreground and shadow segmentation method integrates both intensity and gradient features. Moreover, models of background, shadow, and gradient information are updated adaptively for nonstationary background processes. Experimental results show that the proposed approach can accurately detect moving objects and their cast shadows even in monocular grayscale video sequences.     

Pearson-based mixture model for color object tracking

To track objects in video sequences, many studies have been done to characterize the target with respect to its color distribution. Most often, the Gaussian mixture model (GMM) is used to represent the object color density. In this paper, we propose to extend the normality assumption to more general families of distributions issued from the Pearson’s system. Precisely, we propose a method called Pearson mixture model (PMM), used in conjunction with Gaussian copula, which is dynamically updated to adapt itself to the appearance change of the object during the sequence. This model is combined with Kalman filtering to predict the position of the object in the next frame. Experimental results on gray-level and color video sequences show tracking improvements compared to classical GMM. Especially, the PMM seems robust to illumination variations, pose and scale changes, and also to partial occlusions, but its computing time is higher than the computing time of GMM.     

Global Illumination Shadow Layers

Computer graphics artists often resort to compositing to rework light effects in a synthetic image without requiring a new render. Shadows are primary subjects of artistic manipulation as they carry important stylistic information while our perception is tolerant with their editing. In this paper we formalize the notion of global shadow, generalizing direct shadow found in previous work to a global illumination context. We define an object's shadow layer as the difference between two altered renders of the scene. A shadow layer contains the radiance lost on the camera film because of a given object. We translate this definition in the theoretical framework of Monte‐Carlo integration, obtaining a concise expression of the shadow layer. Building on it, we propose a path tracing algorithm that renders both the original image and any number of shadow layers in a single pass: the user may choose to separate shadows on a per‐object and per‐light basis, enabling intuitive and decoupled edits.     

视频图像中运动目标的阴影消除法

针对在视频序列图像中运动目标的阴影造成运动目标的物理变形,影响运动目标的检测与跟踪;同时,由于目前常用的阴影检测算法,总是要先检测到阴影,再分割阴影,需两步才能达到消除阴影目的,难以达到实时检测要求。为此,提出了一种基于视频图像快速阴影消除方法。该方法通过分析阴影与背景的HSV彩色空间中的特性,利用阴影与运动目标在H,S,V三个分量中的不同特点,计算其相应的阈值,运用该阈值进行分割并消除阴影,能在视频序列图像中一次性检测到运动目标的同时就消除阴影。实验表明,在户外条件下,该算法是准确有效的。     

Light source position and reflectance estimation from a single view without the distant illumination assumption

Several techniques have been developed for recovering reflectance properties of real surfaces under unknown illumination. However, in most cases, those techniques assume that the light sources are located at infinity, which cannot be applied safely to, for example, reflectance modeling of indoor environments. In this paper, we propose two types of methods to estimate the surface reflectance property of an object, as well as the position of a light source from a single view without the distant illumination assumption, thus relaxing the conditions in the previous methods. Given a real image and a 3D geometric model of an object with specular reflection as inputs, the first method estimates the light source position by fitting to the Lambertian diffuse component, while separating the specular and diffuse components by using an iterative relaxation scheme. Our second method extends that first method by using as input a specular component image, which is acquired by analyzing multiple polarization images taken from a single view, thus removing its constraints on the diffuse reflectance property. This method simultaneously recovers the reflectance properties and the light source positions by optimizing the linearity of a log-transformed Torrance-Sparrow model. By estimating the object's reflectance property and the light source position, we can freely generate synthetic images of the target object under arbitrary lighting conditions with not only source direction modification but also source-surface distance modification. Experimental results show the accuracy of our estimation framework.     

Robust object tracking with background-weighted local kernels

Object tracking is critical to visual surveillance, activity analysis and event/gesture recognition. The major issues to be addressed in visual tracking are illumination changes, occlusion, appearance and scale variations. In this paper, we propose a weighted fragment based approach that tackles partial occlusion. The weights are derived from the difference between the fragment and background colors. Further, a fast and yet stable model updation method is described. We also demonstrate how edge information can be merged into the mean shift framework without having to use a joint histogram. This is used for tracking objects of varying sizes. Ideas presented here are computationally simple enough to be executed in real-time and can be directly extended to a multiple object tracking system.     

视频监控系统中运动目标检测的阴影去除方法

在分析了阴影产生的原因及视觉特性的基础上,针对固定场景的运动目标检测问题,提出了一种具有实际应用价值的阴影检测与去除方法。该方法通过背景建模和背景差分方法分割出运动目标及其可能存在的阴影,采用光照评估方法判断阴影是否存在,进而运用多梯度分析和二值图像的快速聚类算法检测出阴影,去除阴影并分割运动目标。实验验证了该方法的有效性。