Registration based on projective reconstruction technique for augmented reality systems

In AR systems, registration is one of the most difficult problems currently limiting their application. In this paper, we propose a simple registration method using projective reconstruction. This method consists of two steps: embedding and tracking. Embedding involves specifying four points to build the world coordinate system on which a virtual object will be superimposed. In tracking, a projective reconstruction technique is used to track these four specified points to compute the model view transformation for augmentation. This method is simple, as only four points need to be specified at the embedding stage and the virtual object can then be easily augmented onto a real scene from a video sequence. In addition, it can be extended to a scenario using the projective matrix that has been obtained from previous registration results using the same AR system. The proposed method has three advantages: 1) it is fast because the linear least square method can be used to estimate the related matrix in the algorithm and it is not necessary to calculate the fundamental matrix in the extended case. 2) A virtual object can still be superimposed on a related area even if some parts of the specified area are occluded during the whole process. 3) This method is robust because it remains effective even when not all the reference points are detected during the whole process, as long as at least six pairs of related reference points correspondences can be found. Some experiments have been conducted to validate the performance of the proposed method.     

A generalized registration method for augmented reality systems

In augmented reality systems, registration is one of the most difficult problems currently limiting their applications. In this paper, we propose a generalized registration method using projective reconstruction technique in computer vision. This registration method is composed of embedding and tracking. Embedding involves specifying four points to build the world coordinate system on which a virtual object will be superimposed. In this stage, any arbitrary two unrelated images or any 3×4 projective matrices with rank 3 can be used to calculate the 3D pseudo-projective coordinates of the four specified points. In the tracking process, these 3D pseudo-projective coordinates are used to track the four specified points to compute the registration matrix for augmentation. The proposed method is simple, as only four points need to be specified at the embedding stage, and the virtual object can then be easily augmented onto a real scene from a video sequence. One advantage is that the virtual objects can still be superimposed on the specified regions even when the regions are occluded in the video sequence. Another advantage of the proposed method is that the registration errors can be adjusted in real-time to ensure that they are less than certain thresholds that have been specified at the initial embedding stage. Several experiments have been conducted to validate the performance of the proposed generalized method.     

一种基于视沉的增强现实三维注册算法

为了提高增强现实系统三维注册的精度和效率，提出了一种基于计算机视觉的增强现实三维注射算法，并已将其用于采用透射式头盔显示器的增强现实系统中，该算法有如下特点：（1）构架简单，实用性强，一般情况下只需4个平面标志物就可实现三维注册；（2）工作范围大，甚至可以应用到室外的增强现实系统中；（30数值求解过程最线性过程，误差小，可以满足增强现实系统高精度三维注册的要求，另外，应用这种算法所需要作的图象处理计算量很小，其典型系统包括一个彩色CCD和几个不同颜色的标志点，由于很容易获得标志点的对应像面位置，而且不需要计算图象对，因此实时性好，是一般图形工作站和PC机上的增强现实系统进行实时注册的算法基础。     

基于投影技术的增强现实注册方法

注册技术是实现增强现实应用系统的关键问题之一。实现了一种利用投影技术改进了的注册方法。这种注册方法主要包含4个步骤：（1）检测标识和特征点的位置;（2）利用投影技术跟踪摄像机;（3）计算模型视图矩阵;（4）绘制虚拟物体。在初始阶段,指定4个点建立世界坐标系,虚拟物体将会被叠加到这个坐标系上;在注册过程中,利用投影技术跟踪特征点的位置,并计算用于增强的模型视图转换。这种注册方法可以为一般的室内增强现实系统提供所需的精确度和实时性。     

Marker-less registration based on template tracking for augmented reality

Accurate 3D registration is a key issue in the Augmented Reality (AR) applications, particularly where are no markers placed manually. In this paper, an efficient markerless registration algorithm is presented for both outdoor and indoor AR system. This algorithm first calculates the correspondences among frames using fixed region tracking, and then estimates the motion parameters on projective transformation following the homography of the tracked region. To achieve the illumination insensitive tracking, the illumination parameters are solved jointly with motion parameters in each step. Based on the perspective motion parameters of the tracked region, the 3D registration, the camera’s pose and position, can be calculated with calibrated intrinsic parameters. A marker-less AR system is described using this algorithm, and the system architecture and working flow are also proposed. Experimental results with comparison quantitatively demonstrate the correctness of the theoretical analysis and the robustness of the registration algorithm.

Natural feature tracking for augmented reality

Natural scene features stabilize and extend the tracking range of augmented reality (AR) pose-tracking systems. We develop robust computer vision methods to detect and track natural features in video images. Point and region features are automatically and adaptively selected for properties that lead to robust tracking. A multistage tracking algorithm produces accurate motion estimates, and the entire system operates in a closed-loop that stabilizes its performance and accuracy. We present demonstrations of the benefits of using tracked natural features for AR applications that illustrate direct scene annotation, pose stabilization, and extendible tracking range. Our system represents a step toward integrating vision with graphics to produce robust wide-area augmented realities     

A 3D GIS-based interactive registration mechanism for outdoor augmented reality system

Registering virtual objects with reference to their corresponding real-world objects plays a key role in augmented reality (AR) system. Although there have been a lot of work on using vision-based method to perform registration for indoor AR system, it is very difficult to apply such registration method for outdoor AR system due to the inability to modify the objects in outdoor environment and the huge range of working area. 3D Geographic Information System (GIS) is capable of providing an outdoor virtual geographic environment where users are located at, which may provide users with a corresponding virtual object for the one in the physical world. In this study, a 3D GIS-based registration mechanism is proposed for outdoor AR system. Specifically, an easy-use interactive method for precise registration was developed to improve the performance of the registration. To implement the registration mechanism, an outdoor AR system built upon 3D GIS was developed, named Augmented Reality Geographical Information System (ARGIS). ARGIS has the capability of performing precise registration in outdoor environment without using traditional vision tracking method, which thus enables users to arbitrarily manipulate the system. A prototype was developed to conduct experiment on the campus of Peking University, Beijing, China to test the proposed registration mechanism. The experiment shows that the developed registration mechanism is feasible and efficient in the outdoor environment. The ARGIS is expected to enrich the applications of outdoor AR system, including but not limited to underground facility mapping, emergency rescue and urban planning.     

基于自然场景在线学习的跟踪注册技术

三维注册是移动增强现实的关键技术之一,提出了一种在线学习的跟踪注册方法,能够精确地对自然场景进行跟踪注册.该方法首先改进SURF（speeded up robust features）描述符匹配方法,提高初始注册矩阵的正确性;然后,通过对场景进行有效的在线学习,提高注册精度;最后,利用前一帧的注册矩阵快速恢复已丢失的关键点,以提高注册的速度.实验结果表明,该方法能够较为流畅地对视频帧进行跟踪,并能保持较好的注册精度.     

使用三角网的户外增强现实注册方式的研究

增强现实的关键技术是进行三维注册,即由真实场景计算虚实结合的参数。为扩大增强现实在户外的应用,提出了在真实场景主要物体上增加表面3D网格的方法来减少虚实结合参数计算的复杂程度。该方法既不需要增强现实室内应用开发所应用的标记,也无需目前户外应用所依赖的传感仪器。以户外地形为例,通过对视线与地形网格交点的计算快速得到以往需要两幅图像联合处理才能得到的三维信息并进行快速的虚实合成。该方法为户外增强现实系统关键技术的研究提供了一个新的思路。     

Harmonic rendering for visual coherence on mobile outdoor AR environment

Rapid developments in augmented reality (AR) and related technologies have led to increasing interest in immersive content. AR environments are created by combining virtual 3D models with a real-world video background. It is important to merge these two worlds seamlessly if users are to enjoy AR applications, but, all too often, the illumination and shading of virtual objects is not consider the real world lighting condition or does not match that of nearby real objects. In addition, visual artifacts produced when blending real and virtual objects further limit realism. In this paper, we propose a harmonic rendering technique that minimizes the visual discrepancy between the real and virtual environments to maintain visual coherence in outdoor AR. To do this, we introduce a method of estimating and approximating the Sun’s position and the sunlight direction to estimate the real sunlight intensity, as this is the most significant illumination source in outdoor AR and it provides a more realistic lighting environment for such content, reducing the mismatch between real and virtual objects.

     

改进TLD与ORB的AR系统长时跟踪注册方法

针对增强现实系统长时跟踪注册需求以及三维跟踪注册时待注册区域尺度变化、光照变化、内外旋转等复杂环境易导致注册失败问题,提出改进TLD跟踪及ORB特征检测的增强现实长时三维注册方法。通过采用TLD长时鲁棒跟踪算法对待注册区域跟踪,并利用待注册位置自适应动态检测策略对TLD算法跟踪-学习-检测的结构机制的实时性和精确度进行优化;利用ORB算法对待注册位置进行特征检测及三维注册矩阵计算;将虚拟信息叠加至待注册位置。选取长时视频序列进行跟踪、检测及注册仿真实验。实验结果表明,改进TLD长时跟踪算法相比TLD算法在OPE精确度方面提升约5%,对待注册位置的跟踪速率提升约2倍,使得AR系统在复杂环境下的长时跟踪注册具有更好的稳定性和鲁棒性。     

基于ARToolKit的地下管网增强现实系统研究

增强现实是把计算机生成的虚拟图像或其它信息叠加到用户所看到的真实世界中的一种技术,是虚拟现实领域的一个难点热点。ARToolKit为增强现实技术的应用提供了一种方便快捷的开发工具,但其不足之处在于它是基于计算机视觉的注册方式,不适合户外增强现实系统的开发,需要做进一步的改进。论文利用ARToolKit在室内实现了管网三维可视化的增强显示,取得了满意的效果;根据户外的实际,提出了以全球定位系统载波相位差分(GPSRTK)和惯性导航系统(INS)相结合进行精确定位、以电子罗盘和倾角计相结合测定视线方向的混合注册方法;对利用ARToolKit建立户外地下管网增强现实系统的框架及功能作了进一步的探讨。     

An automatic image-to-DEM alignment approach for annotating mountains pictures on a smartphone

Augmented reality (AR) technologies enhance users knowledge of their immediate surroundings by presenting contextualized and spatially relevant information. This augmentation is enabled by the automatic estimation of the device’s pose with respect to the environment. Many simple AR applications for smartphones are well served by the pose estimate provided by GPS and inertial sensors, while other demand an higher accuracy which can be obtained with vision-based methods. The application presented in this paper, which aims to augment pictures of mountainous landscapes with geo-referenced data, belongs to the latter category. Our application is based on a novel approach for image-to-world registration which jointly relies on inertial and visual sensors. In a nutshell, first GPS and inertial sensors are used to compute a rough estimate of the device position and pose, then visual data are employed to refine it. Specifically, a learning-based edge detection algorithm is used to extract mountain profiles from the picture of interest. Then, a novel registration algorithm based on a robust optimization framework is employed to align the computed profiles to synthetic ones obtained from Digital Elevation Models. Our experiments, conducted on a novel dataset of manually aligned pictures which is made publicly available, demonstrate that the proposed registration method guarantees an increased accuracy with respect to competing methods and it is computationally efficient when implemented on a smartphone.     

基于SURF与光流法的增强现实跟踪注册

提出一种基于SURF与光流法相结合的增强现实局部跟踪注册方法。采用光流法对移动对象区域进行跟踪,利用SURF算法仿射、尺度不变性及运算速度快的优点对该区域进行特征提取与匹配,利用相邻帧之间特征点的匹配关系求得三维注册矩阵,在保持注册精确性的同时降低了系统运算时间。实验结果表明该方法达到了实时跟踪与准确注册的效果,并且在环境变化时保持了较好的鲁棒性。     

Pose estimation for planar structures

We address the registration problem for interactive AR applications. Such applications require a real-time registration process. Although the registration problem has received a lot of attention in the computer vision community, it's far from being solved. Ideally, an AR system should work in all environments without the need to prepare the scene ahead of time, and users should be able to walk anywhere they want. In the past, several AR systems have achieved accurate and fast tracking and registration, putting dots over objects and tracking the dots with a camera. We can also achieve registration by identifying features in the scene that we can carefully measure for real-world coordinates. However, such methods restrict the system's flexibility. Hence, we need to investigate registration methods that work in unprepared environments and reduce the need to know the objects' geometry in the scene. We propose an efficient solution to real-time camera tracking for scenes that contain planar structures. We can consider many types of scene with our method. We show that our system is reliable and we can use it for real-time applications. We also present results demonstrating real-time camera tracking on indoor and outdoor scenes.     

Multithreaded Hybrid Feature Tracking for Markerless Augmented Reality

We describe a novel markerless camera tracking approach and user interaction methodology for augmented reality (AR) on unprepared tabletop environments. We propose a real-time system architecture that combines two types of feature tracking. Distinctive image features of the scene are detected and tracked frame-to-frame by computing optical flow. In order to achieve real-time performance, multiple operations are processed in a synchronized multi-threaded manner: capturing a video frame, tracking features using optical flow, detecting distinctive invariant features, and rendering an output frame. We also introduce user interaction methodology for establishing a global coordinate system and for placing virtual objects in the AR environment by tracking a user's outstretched hand and estimating a camera pose relative to it. We evaluate the speed and accuracy of our hybrid feature tracking approach, and demonstrate a proof-of-concept application for enabling AR in unprepared tabletop environments, using bare hands for interaction.     

Abnormality detection using low-level co-occurring events

We propose in this paper a method for behavior modeling and abnormal events detection which uses low-level features. In conventional object-based approaches, objects are identified, classified, and tracked to locate those with suspicious behavior. We proceed directly with event characterization and behavior modeling using low-level features. We first learn statistics about co-occurring events in a spatio-temporal volume in order to build the normal behavior model, called the co-occurrence matrix. The notion of co-occurring events is defined using mutual information between motion labels sequences. Then, in the second phase, the co-occurrence matrix is used as a potential function in a Markov random field framework to describe, as the video streams in, the probability of observing new volumes of activity. The co-occurrence matrix is thus used for detecting moving objects whose behavior differs from the ones observed during the training phase. Interestingly, the Markov random field distribution implicitly accounts for speed, direction, as well as the average size of the objects without any higher-level intervention. Furthermore, when the spatio-temporal volume is sufficiently large, the co-occurrence distribution contains the average normal path followed by moving objects. Our method has been tested on various indoor and outdoor videos representing various challenges.     

基于SLAM的虚实配准方法研究

研究和实现了一种基于SLAM技术的虚实配准方法,通过将SLAM分离为定位和构图,分别利用改进的pf做位姿估计和ukf进行位置估计,进而实现整个场景内目标的位姿计算,最终完成虚实配准。一定程度上解决了传统方法中存在的视域受限问题,并同时支持户内和户外应用,提高了增强现实系统的可用性。     

视频图象序列的实时镶嵌技术研究

阐述了一种快速而高效的由视频图象或视频图象序列生成全景的配准方法,为了估计图象配准的校正参数,该方法计算伪运动矢量,这些伪运动矢量是光流在每一选定象素处的粗略估计,使用方法,实现了一个在低价PC上就能实时创建和显示全景图像的软件。     

基于二维透射变换的视频全景图拼接算法

视频序列全景图拼接技术是视频图像分析的关键技术之一。针对帧间运动模型为二维投影变换的常见视频序列,提出了一种全景图拼接算法.该算法首先采用改进的基于RANSAC的特征点匹配算法对相邻帧的运动参数进行估计,接着采用直接法对各帧与全景图间的运动参数进行精确调整,最后运用自适应中值滤波绘制出全景图。实验结果表明,该方法能够生成较高质量的全景图,基本能实现无缝连接。