Interactive 3D content insertion in images for multimedia applications

This article addresses the problem of creating interactive mixed reality applications where virtual objects interact in images of real world scenarios. This is relevant to create games and architectural or space planning applications that interact with visual elements in the images such as walls, floors and empty spaces. These scenarios are intended to be captured by the users with regular cameras or using previously taken photographs. Introducing virtual objects in photographs presents several challenges, such as pose estimation and the creation of a visually correct interaction between virtual objects and the boundaries of the scene. The two main research questions addressed in this article include, the study of the feasibility of creating interactive augmented reality (AR) applications where virtual objects interact in a real world scenario using the image detected high-level features and, also, verifying if untrained users are capable and motivated enough to perform AR initialization steps. The proposed system detects the scene automatically from an image with additional features obtained using basic annotations from the user. This operation is significantly simple to accommodate the needs of non-expert users. The system analyzes one or more photos captured by the user and detects high-level features such as vanishing points, floor and scene orientation. Using these features it will be possible to create mixed and augmented reality applications where the user interactively introduces virtual objects that blend with the picture in real time and respond to the physical environment. To validate the solution several system tests are described and compared using available external image datasets.     

一种基于多尺度小波分解的图象镶嵌技术

为了获得一种具有艺术视觉效果的镶嵌图象,提出了一种基于多尺度小波分解的图象镶嵌技术,该技术首先对原始图象的各子块区域和图象库中的每一幅图象进行多尺度小波分解;然后逐层计算各图象小波分解系数的标准方差和它们之间的相似距离,并据此从图象库中选取与原始图象中的各子块区域最佳匹配的贴图,再将其镶嵌到原始图象中的对应区域;最后对贴图进行逐像素的颜色校正,使贴图的颜色与原始图象尽可能一致.同时,根据人眼观察某个区域时往往通过取整或将细节取平均来得到一个总体强度效果这一视觉特性,使最终的镶嵌图象具有在近处看到的是各个贴图的内容,而在远处观看则是原始图象的总体轮廓的视觉效果.利用计算机来自动地生成这种镶嵌图象的实验结果表明,该方法是有效的.     

Detecting image orientation based on low-level visual content

Accurately and automatically detecting image orientation is of great importance in intelligent image processing. In this paper, we present automatic image orientation detection algorithms based on both the luminance (structural) and chrominance (color) low-level content features. The statistical learning support vector machines (SVMs) are used in our approach as the classifiers. The different sources of the extracted image features, as well as the binary classification nature of SVM, require our system to be able to integrate the outputs from multiple classifiers. Both static combiner (averaging) and trainable combiner (also based on SVMs) are proposed and evaluated in this work. Furthermore, two rejection options (regular and re-enforced ambiguity rejections) are employed to improve orientation detection accuracy by sieving out images with low confidence values during the classification. Large amounts of experiments have been conducted on a database of more than 14,000 images to validate our approaches. Discussions and future directions for this work are also addressed at the end of the paper.     

Invertible Apertured Orientation Filters in Image Analysis

A unitary approach for locally apertured orientation analysis of 2D and 3D scalar images is proposed. The size of the local aperture (the scale) needed for the orientation representation induces in general a lost of spatial acuity, or blur. Our construction permits a compensation of the blur by a reconstruction procedure. For this purpose, a special scale-dependent orientation bundle (map of the visual space into function of both position and orientation) is build from the local Gaussian-derivatives jet of a scalar image. In this construction there is an invertible relation between the orientation bundle and the original image. This invertible transformation is used to regain the original acuity in the spatial domain after analyzing orientation features at any given scale.The approach turns out to be highly effective for the detection of elongated structures and for removal of elongated artifacts in 2D images.     

基于边缘融合的多图像直线提取算法

该文提出了一种新的基于边缘融合的多图像直线提取算法，该算法首先对所有多图像分别提取边缘，然后通过将多图像对应的梯度方向相似的、相邻近的边缘编组，以形成融合的直线支持区域，最后基于最小二乘的方法从直线支持区域拟合出直线段，并提取直线属性．该文提出的算法把边缘的相位信息作为融合要素，能够从复杂的图像中提取出相当低对比度的直线，并且通过分析多图像边缘之间的特性，提出相位压缩和融合编组方案，不仅可以充分融合互补信息还有效降低了算法复杂度．真实的遥感图像直线提取结果表明了该文算法的有效性．     

基于Zernike色度分布矩的彩色图像检索算法

以对立色度空间及Zernike矩理论为基础,提出一种基于Zernike色度分布矩的彩色图像检索方法。该方法首先将彩色图像从三维RGB颜色空间转换到二维对立色度空间,以获得图像像素的色差度量值。然后结合Zernike矩相关知识,在二维色度空间中计算出各像素的Zernike色度分布矩,最后将Zernike色度分布矩作为图像颜色特征,并进行彩色图像检索。仿真实验表明,该算法能够准确高效地查找出用户所需内容的彩色图像,并且具有较好的查准率和查全率。     

Stereo-based palmprint recognition in various 3D postures

In order to increase performance in palmprint recognition systems, various devices are normally used to restrict the movement of the hand. These can cause problems, especially for those users with physical disabilities. They also cause significant hygiene problems in multi-user systems. Recently, studies on palmprint recognition systems have progressed towards the development of unconstrained, contactless and unrestricted background techniques. The most common problem encountered in these studies is the alignment arising from the free movement of the hand. Despite 3D hand-acquisition devices which offer extra recognition features to overcome this problem, the applicability of these devices is low because of their increased cost. In this study, a stereo camera was proposed. Although due to matching problems, it is difficult to achieve precise, distinct feature extraction in the unrestricted 3D environment used for palmprint recognition, the orientation of the hand in 3D space can be determined by obtaining depth information. In this study, the depth information was extracted by using the binocular stereo approach. First, the orientation of the hand was estimated by fitting a surface model associated with the eigenvectors of the depth information. Pose correction was then accomplished by establishing a relationship between the orientation and the images. The pose correction greatly relieved the perspective distortion that usually occurs within the various poses of the hands. Next, the region of interest was determined by performing segmentation on the corrected images using the Active Appearance Model (AAM). The palmprint features were then extracted via Gabor-based Kernel Fisher Discriminant Analysis. In order to demonstrate the performance of the proposed approach, a new dataset was compiled from stereo images within various scenarios collected from 138 different individuals. As a result of these experimental studies, the EER values, especially on the images captured from different hand orientations in 3D, were reduced from around 14–0.75%. With the help of this suggested approach, the palmprint recognition system was transformed into a more portable form by removing the closed-box mechanisms and equipment restricting movement of the hand. This system can automatically perform pose estimation, hand segmentation and recognition processes without any special intervention.     

基于多分辨率生成对抗网络的空间数据不确定性重建方法

在空间数据不确定性重建领域,多点统计法（MPS）得到了广泛的应用,但由于计算成本较高,其适用性受到了影响。通过使用金字塔结构的全卷积生成对抗网络（GAN）模型学习不同分辨率的训练图像,提出了一种基于多分辨率GAN模型的空间数据重建方法。该方法从高分辨率训练图像中捕获细节特征,从低分辨率训练图像中捕获大范围特征,因此该方法重建的图像包含训练图像的全局和局部结构信息,同时保持一定的随机性。把所提算法与MPS中的代表性算法以及应用于空间数据重建的GAN方法进行对比的结果表明,所提方法10次重建的总时间降低了约1 h,其平均孔隙度与训练图像孔隙度的差值降低至0.000 2,并且其变差函数曲线和多点连接性函数（MPC）曲线更接近于训练图像,可见所提算法重建质量更好。     

Recovery of the 3-D location and motion of a rigid object through camera image (An Extended Kalman Filter Approach)

This paper deals with the problem of locating a rigid object and estimating its motion in three dimensions. This involves determining the position and orientation of the object at each instant when an image is captured by a camera, and recovering the motion of the object between consecutive frames.In the implementation scheme used here, a sequence of camera images, digitized at the sample instants, is used as the initial input data. Measurements are made of the locations of certain features (e.g., maximum curvature points of an image contour, corners, edges, etc.) on the 2-D images. To measure the feature locations a matching algorithm is used, which produces correspondences between the features in the image and the object.Using the measured feature locations on the image, an algorithm is developed to solve the location and motion problem. The algorithm is an extended Kalman filter modeled for this application.Department of Electrical Engineering and Alberta Center for Machine Intelligence and Robotics, University of Alberta     

瓷砖图像的纹理特征分类研究

在已有的瓷砖图像分类系统中,仅靠颜色特征和简单的纹理边缘信息只能对无花纹的单色砖或简单花纹的瓷砖进行有效分类,对复杂图案的瓷砖存在识别率低的问题。针对此种情况,结合瓷砖图像的灰度共生矩阵和统计几何特征,将这些特征输入支持向量机进行特征分层分类。采用基于径向基核函数和[K]交叉验证法所得到的最优参数构造支持向量机,解决瓷砖纹理特征具有非线性的分类问题。用瓷砖生产线上采集的大量图像进行实验表明,该方法准确率高,分类效果好。     

Explicit and implicit employment of edge-related information in super-resolving distant faces for recognition

Edges and related features play significant role in discriminating face images. But those features are not enough informative when the face images are captured from a distance (e.g., video surveillance). Traditionally, those features are enhanced by super-resolving low-resolution grayscale face images. In this paper, we demonstrate a superior performance by directly considering such features (continuous gradient value, also known as edginess) in the super-resolution process. Edginess features are extracted using 1-D processing of image. This process is carried out along different directions to obtain partial evidences, which are combined to detect the person’s identity. Here, super-resolution of the face image and its recognition has been performed in sparse domain framework. The explicit usage of edginess feature in the proposed approach shows considerable improvement in both recognition performance as well as computational time, as only the patches related to strong edges are considered for super-resolution. In addition to that, the edginess feature gives improved recognition rate when it is preserved implicitly during super-resolution in grayscale domain.     

基于谱相关性的数字图像真伪鉴别

如何区分逼真的计算机生成图像和真实的自然图像,是数字图像取证领域的一个重要研究方向。提出了一种基于谱间相关性的图像真伪鉴别算法。在单CCD数码成像过程中,每个像素只采集单一颜色值,缺失的颜色值通过颜色滤波阵列插值获得,而基于颜色滤波阵列插值的去马赛克方法会引起彩色图像三个颜色分量之间较高的谱间相关性。算法利用小波变换和标准互相关系数提取颜色组件谱间相关性,作为区分和识别特征。通过在标准图像库上的实验测试,表明所提取特征有效捕获了二类图像的差别,并具有较高的检测率。     

Accurate sensing of scene geo-context via mobile visual localization

Image geo-tagging has drawn a great deal of attention in recent years. The geographic information associated with images can be used to promote potential applications such as location recognition or virtual navigation. In this paper, we propose a novel approach for accurate mobile image geo-tagging in urban areas. The approach is able to provide a comprehensive set of geo-context information based on the current image, including the real location of the camera and the viewing angle, as well as the location of the captured scene. Moreover, the parsed building facades and their geometric structures can also be estimated. First, for the image to be geo-tagged, we perform partial duplicate image retrieval to filter crowd-sourced images capturing the same scene. We then employ the structure-from-motion technique to reconstruct a sparse 3D point cloud of the scene. Meanwhile, the geometric structure of the query image is analyzed to extract building facades. Finally, by combining the reconstructed 3D scene model and the extracted structure information, we can register the camera location and viewing direction to a real-world map. The captured building location and facade orientation are also aligned. The effectiveness of the proposed system is demonstrated by experiment results.     

Tracking 3-D motion from straight lines with trifocal tensors

We present a novel approach to track the position and orientation of a stereo camera using line features in the images. The method combines the strengths of trifocal tensors and Bayesian filtering. The trifocal tensor provides a geometric constraint to lock line features among every three frames. It eliminates the explicit reconstruction of the scene even if the 3-D scene structure is not known. Such a trifocal constraint thus makes the algorithm fast and robust. The twist motion model is applied to further improve its computation efficiency. Another major contribution is that our approach can obtain the 3-D camera motion using as little as 2 line correspondences instead of 13 in the traditional approaches. This makes the approach attractive for realistic applications. The performance of the proposed method has been evaluated using both synthetic and real data with encouraging results. Our algorithm is able to estimate 3-D camera motion in real scenarios accurately having little drifting from an image sequence longer than a 1,000 frames.     

Horizon matching for localizing unordered panoramic images

There is currently an abundance of vision algorithms which, provided with a sequence of images that have been acquired from sufficiently close successive 3D locations, are capable of determining the relative positions of the viewpoints from which the images have been captured. However, very few of these algorithms can cope with unordered image sets. This paper presents an efficient method for recovering the position and orientation parameters corresponding to the viewpoints of a set of panoramic images for which no a priori order information is available, along with certain structure information regarding the imaged environment. The proposed approach assumes that all images have been acquired from a constant height above a planar ground and operates sequentially, employing the Levenshtein distance to deduce the spatial proximity of image viewpoints and thus determine the order in which images should be processed. The Levenshtein distance also provides matches between imaged points, from which their corresponding environment points can be reconstructed. Image matching with the aid of the Levenshtein distance forms the crux of an iterative process that alternates between image localization from multiple reconstructed points and point reconstruction from multiple image projections, until all views have been localized. Periodic refinement of the reconstruction with the aid of bundle adjustment, distributes the reconstruction error among images. The approach is demonstrated on several unordered sets of panoramic images obtained in indoor environments.     

基于图像不变矩和神经网络的零件位姿自动识别

零件位置的自动识别是智能三坐标测量机检测系统中极为关键的技术.根据零件可能的放置方式形成零件的多幅虚拟图像,利用CCD摄像机采集零件的实际图像.将图像不变矩理论应用于图像匹配技术中,以虚拟图像的矩不变量及其对应模式进行BP神经网络的训练,将训练好的网络作为分类器,根据实际采集图像的矩不变量进行零件姿态的识别;由采集图像的不变矩计算零件在像平面内的位置和方向,并经坐标转换得到零件在机器坐标系中的位置和方向.实验结果表明本文零件位置自动识别方法是智能且高效的.     

Transductive cost-sensitive lung cancer image classification

Previous computer-aided lung cancer image classification methods are all cost-blind, which assume that the misdiagnosis (categorizing a cancerous image as a normal one or categorizing a normal image as a cancerous one) costs are equal. In addition, previous methods usually require experienced pathologists to label a large amount of images as training samples. To this end, a novel transductive cost-sensitive method is proposed for lung cancer image classification on needle biopsies specimens, which only requires the pathologist to label a small amount of images. The proposed method analyzes lung cancer images in the following procedures: (i) an image capturing procedure to capture images from the needle biopsies specimens; (ii) a preprocessing procedure to segment the individual cells from the captured images; (iii) a feature extraction procedure to extract features (i.e. shape, color, texture and statistical information) from the obtained individual cells; (iv) a codebook learning procedure to learn a codebook on the extracted features by adopting k-means clustering, which aims to represent each image as a histogram over different codewords; (v) an image classification procedure to predict labels for testing images using the proposed multi-class cost-sensitive Laplacian regularized least squares (mCLRLS). We evaluate the proposed method on a real-image set provided by Bayi Hospital, which contains 271 images including normal ones and four types of cancerous ones (squamous carcinoma, adenocarcinoma, small cell cancer and nuclear atypia). The experimental results demonstrate that the proposed method achieves a lower cancer-misdiagnosis rate and lower total misdiagnosis costs comparing with previous methods, which includes the supervised learning approach (kNN, mcSVM and MCMI-AdaBoost), semi-supervised learning approach (LapRLS) and cost-sensitive approach (CS-SVM). Meanwhile, the experiments also disclose that both transductive and cost-sensitive settings are useful when only a small amount of training images are available.     

Visual graph modeling for scene recognition and mobile robot localization

Image retrieval and categorization may need to consider several types of visual features and spatial information between them (e.g., different point of views of an image). This paper presents a novel approach that exploits an extension of the language modeling approach from information retrieval to the problem of graph-based image retrieval and categorization. Such versatile graph model is needed to represent the multiple points of views of images. A language model is defined on such graphs to handle a fast graph matching. We present the experiments achieved with several instances of the proposed model on two collections of images: one composed of 3,849 touristic images and another composed of 3,633 images captured by a mobile robot. Experimental results show that using visual graph model (VGM) improves the accuracies of the results of the standard language model (LM) and outperforms the Support Vector Machine (SVM) method.     

Flexible depth of field photography

The range of scene depths that appear focused in an image is known as the depth of field (DOF). Conventional cameras are limited by a fundamental trade-off between depth of field and signal-to-noise ratio (SNR). For a dark scene, the aperture of the lens must be opened up to maintain SNR, which causes the DOF to reduce. Also, today's cameras have DOFs that correspond to a single slab that is perpendicular to the optical axis. In this paper, we present an imaging system that enables one to control the DOF in new and powerful ways. Our approach is to vary the position and/or orientation of the image detector during the integration time of a single photograph. Even when the detector motion is very small (tens of microns), a large range of scene depths (several meters) is captured, both in and out of focus. Our prototype camera uses a micro-actuator to translate the detector along the optical axis during image integration. Using this device, we demonstrate four applications of flexible DOF. First, we describe extended DOF where a large depth range is captured with a very wide aperture (low noise) but with nearly depth-independent defocus blur. Deconvolving a captured image with a single blur kernel gives an image with extended DOF and high SNR. Next, we show the capture of images with discontinuous DOFs. For instance, near and far objects can be imaged with sharpness, while objects in between are severely blurred. Third, we show that our camera can capture images with tilted DOFs (Scheimpflug imaging) without tilting the image detector. Finally, we demonstrate how our camera can be used to realize nonplanar DOFs. We believe flexible DOF imaging can open a new creative dimension in photography and lead to new capabilities in scientific imaging, vision, and graphics.