A 3D GIS-based interactive registration mechanism for outdoor augmented reality system

Registering virtual objects with reference to their corresponding real-world objects plays a key role in augmented reality (AR) system. Although there have been a lot of work on using vision-based method to perform registration for indoor AR system, it is very difficult to apply such registration method for outdoor AR system due to the inability to modify the objects in outdoor environment and the huge range of working area. 3D Geographic Information System (GIS) is capable of providing an outdoor virtual geographic environment where users are located at, which may provide users with a corresponding virtual object for the one in the physical world. In this study, a 3D GIS-based registration mechanism is proposed for outdoor AR system. Specifically, an easy-use interactive method for precise registration was developed to improve the performance of the registration. To implement the registration mechanism, an outdoor AR system built upon 3D GIS was developed, named Augmented Reality Geographical Information System (ARGIS). ARGIS has the capability of performing precise registration in outdoor environment without using traditional vision tracking method, which thus enables users to arbitrarily manipulate the system. A prototype was developed to conduct experiment on the campus of Peking University, Beijing, China to test the proposed registration mechanism. The experiment shows that the developed registration mechanism is feasible and efficient in the outdoor environment. The ARGIS is expected to enrich the applications of outdoor AR system, including but not limited to underground facility mapping, emergency rescue and urban planning.     

A systematic framework for on‐line calibration of a head‐mounted projection display for augmented‐reality systems

Abstract— Augmented reality (AR) is a technology in which computer‐generated virtual images are dynamically superimposed upon a real‐world scene to enhance a user's perceptions of the physical environment. A successful AR system requires that the overlaid digital information be aligned with the user's real‐world senses — a process known as registration. An accurate registration process requires the knowledge of both the intrinsic and extrinsic parameters of the viewing device and these parameters form the viewing and projection transformations for creating the simulations of virtual images. In our previous work, an easy off‐line calibration method in which an image‐based automatic matching method was used to establish the world‐to‐image correspondences was presented, and it is able to achieve subpixel accuracy. However, this off‐line method yields accurate registration only when a user's eye placements relative to the display device coincides with locations established during the offline calibration process. A likely deviation of eye placements, for instance, due to helmet slippage or user‐dependent factors such as interpupillary distance, will lead to misregistration. In this paper, a systematic on‐line calibration framework to refine the off‐line calibration results and to account for user‐dependent factors is presented. Specifically, based on an equivalent viewing projection model, a six‐parameter on‐line calibration method to refine the user‐dependent parameters in the viewing transformations is presented. Calibration procedures and results as well as evaluation experiments are described in detail. The evaluation experiments demonstrate the improvement of the registration accuracy.     

Registration based on projective reconstruction technique for augmented reality systems

In AR systems, registration is one of the most difficult problems currently limiting their application. In this paper, we propose a simple registration method using projective reconstruction. This method consists of two steps: embedding and tracking. Embedding involves specifying four points to build the world coordinate system on which a virtual object will be superimposed. In tracking, a projective reconstruction technique is used to track these four specified points to compute the model view transformation for augmentation. This method is simple, as only four points need to be specified at the embedding stage and the virtual object can then be easily augmented onto a real scene from a video sequence. In addition, it can be extended to a scenario using the projective matrix that has been obtained from previous registration results using the same AR system. The proposed method has three advantages: 1) it is fast because the linear least square method can be used to estimate the related matrix in the algorithm and it is not necessary to calculate the fundamental matrix in the extended case. 2) A virtual object can still be superimposed on a related area even if some parts of the specified area are occluded during the whole process. 3) This method is robust because it remains effective even when not all the reference points are detected during the whole process, as long as at least six pairs of related reference points correspondences can be found. Some experiments have been conducted to validate the performance of the proposed method.     

增强现实:特点、关键技术和应用

增强现实(Augmented Reality，AR)技术将计算机生成的虚拟对象与真实世界进行融合，构造出虚实结合的虚拟空间增强现实系统有三个特点：真实和虚拟结合，实时交互，真实和虚拟对象进行定位构造一个AR系统要解决许多关键技术，包括显示技术，跟踪和定位技术，界面技术以及标定技术另外AR系统还要充分考虑人的因素AR技术在许多方面都有潜在的应用价值，如医疗，机械维护和修理，注释，协同，商业以及军事应用等本文对这些问题进行论述，最后讨论为使AR广泛使用需要解决的一些问题。     

增强现实综述

增强现实(augmented reality，AR)技术可以将虚拟的物体合并到现实场景中，并能支持用户与其进行交互，它已经成为虚拟现实研究中的一个重要领域，也是人机界面技术发展的一个重要方向。为了使人们对其有所了解，该文首先概略描述了这个领域的主要研究内容和进展情况，并详细介绍了增强现实中的支撑技术、开发工具和相关理论；然后针对当前AR应用的现状，分析了实现中的难点问题，并给出了与AR普及应用密切相关的一些系统框架和开发平台的描述，最后介绍了几个典型的AR应用实例。     

Deep learning-based smart task assistance in wearable augmented reality

Wearable augmented reality (AR) smart glasses have been utilized in various applications such as training, maintenance, and collaboration. However, most previous research on wearable AR technology did not effectively supported situation-aware task assistance because of AR marker-based static visualization and registration. In this study, a smart and user-centric task assistance method is proposed, which combines deep learning-based object detection and instance segmentation with wearable AR technology to provide more effective visual guidance with less cognitive load. In particular, instance segmentation using the Mask R-CNN and markerless AR are combined to overlay the 3D spatial mapping of an actual object onto its surrounding real environment. In addition, 3D spatial information with instance segmentation is used to provide 3D task guidance and navigation, which helps the user to more easily identify and understand physical objects while moving around in the physical environment. Furthermore, 2.5D or 3D replicas support the 3D annotation and collaboration between different workers without predefined 3D models. Therefore, the user can perform more realistic manufacturing tasks in dynamic environments. To verify the usability and usefulness of the proposed method, we performed quantitative and qualitative analyses by conducting two user studies: 1) matching a virtual object to a real object in a real environment, and 2) performing a realistic task, that is, the maintenance and inspection of a 3D printer. We also implemented several viable applications supporting task assistance using the proposed deep learning-based task assistance in wearable AR.     

增强现实三维注册中标识点匹配算法研究与改进

基于标识点的增强现实系统中,计算机生成的虚拟物体需要正确地注册到相应的标识点上构成对现实环境的增强.针对增强现实三维注册过程中传统算法标识点匹配过程中误识率较高,匹配不够精确的不足,提出了将相对误差法进行改进实现增强现实中的三维注册,同时提出了将基于颜色直方图的图像匹配原理引入增强现实的三维注册过程中,通过实验验证与分析,改进后的匹配算法与传统算法相比减少了标识的误识率,提高了注册精度,已成功应用于增强现实系统的三维注册过程中.     

基于视觉的增强现实技术研究综述

增强现实技术是当前计算机视觉领域中的热点问题之一,本文系统综述了基于视觉的增强现实 技术的研究现状．首先,对基于视觉的增强现实技术的研究现状进行了概述;其次,详细地阐述了该领域目 前出现的新技术,包括三维注册技术、摄像机标定技术、跟踪技术、虚实光照的一致性以及AR 实空间建模技 术,继而论述了该领域在研究中存在的问题,并对该领域中需要进一步解决的问题进行了讨论;最后,对增 强现实技术的进一步研究进行了展望．     

Information at a glance [augmented reality user interfaces]

What if we could visualize and interact with information directly in the context of our surroundings? Our research group is exploring how augmented reality (AR) could someday make this possible. AR integrates a complementary virtual world with the physical world-for example, by using head-tracked see-through head-worn displays to overlay graphics on what we see. Instead of looking back and forth between the real world and a PDA, we look directly at the real world and the virtual information overlaid on it. At the heart of this approach is context-aware computing, computing systems that are sensitive to the context in which they operate, ranging from human relationships to physical location. For example, information might be tied to specific locations within a global, Earth-centered, coordinate system. How can we design effective mobile AR user interfaces? We've been trying to answer this question in part by developing experimental AR research prototypes. In AR, as in work on information visualization using desktop technologies, the amount of information available can far exceed what a system can legibly display at a given time, necessitating information filtering. Julier et al. (2000) have developed information filtering techniques for AR that depend on the user's goals, object importance, and proximity. We assume that a system can accomplish information filtering of this sort and that our system is displaying everything it should.     

Augmented reality-based robot teleoperation system using RGB-D imaging and attitude teaching device

Augmented reality (AR)-based programming using the demonstration method has been widely studied. However, studies on AR-based programming for remote robots are lacking because of the limitation of human–computer interaction. This paper proposes an AR-based robot teleoperation system and method using RGB-D imaging and an attitude teaching device. By sending the color and depth images of the remote robot environment to the local side, the operators can complete the teleoperation of the robot at the local side. First, the operators select key positions on the motion path of the robot endpoint from color images via a mouse, and the computer calculates the 3D coordinates of these key points in the robot base coordinate system to complete the position teaching process. In the robot attitude teaching process, the AR technology is used to superimpose the virtual robot model onto the color images of the robot teleoperation environment, so as to make the virtual robot endpoint to move along the teaching path. An operator can use the portable attitude teaching device designed in this study to control the robot movement parameters, such as the attitude and motion speed, during the movement of the virtual robot. After the position and attitude teaching processes, the robot movement trajectory can be generated. To make the base coordinate system of the virtual model consistent with that of the physical robot, we propose an online AR registration method, which does not require manually placing the AR registration marker. The proposed AR-based robot teleoperation system can quickly and easily complete robot teleoperation at the local side.     

基于ARToolKit的地下管网增强现实系统研究

增强现实是把计算机生成的虚拟图像或其它信息叠加到用户所看到的真实世界中的一种技术,是虚拟现实领域的一个难点热点。ARToolKit为增强现实技术的应用提供了一种方便快捷的开发工具,但其不足之处在于它是基于计算机视觉的注册方式,不适合户外增强现实系统的开发,需要做进一步的改进。论文利用ARToolKit在室内实现了管网三维可视化的增强显示,取得了满意的效果;根据户外的实际,提出了以全球定位系统载波相位差分(GPSRTK)和惯性导航系统(INS)相结合进行精确定位、以电子罗盘和倾角计相结合测定视线方向的混合注册方法;对利用ARToolKit建立户外地下管网增强现实系统的框架及功能作了进一步的探讨。     

基于增强现实的虚拟城市规划关键技术研究

增强现实是虚拟现实技术的重要分支,已经被成功的应用到众多领域.探讨了增强现实应用于虚拟城市规划的可行性,并给出了一种新颖的基于增强现实技术的城市规划方法.它可以满足城市规划对虚实交互、人机交互的要求,充分体现出增强现实技术在城市规划应用中的优势.介绍了系统结构与工作流程,着重分析了所采用的基于标识的虚实注册方法,并给出了运行结果.     

增强现实技术综述

增强现实技术是将计算机渲染生成的虚拟场景与真实世界中的场景无缝融合起来的一种技术,它通过视频显示设备将虚实融合的场景呈现给用户,使人们与计算机之间的交互更加的自然,同时具有广泛的应用前景,因此成为近年来的一个研究热点。随着跟踪注册技术的进步、计算机性能的飞速发展、深度摄像机的普及,以及Light Field投影技术在增强现实中的应用,增强现实技术逐渐成为下一代人机交互的发展方向。该文章首先概述了增强现实的主要研究内容和发展情况,并详细介绍了增强现实的关键技术、开发工具,然后分类概述了增强现实应用案例。     

A fast object registration method for augmented reality assembly with simultaneous determination of multiple 2D-3D correspondences

A fast and robust object registration is one of the fundamental steps of an augmented reality (AR) assembly guidance system. To manage the registration of texture-less objects from monocular images, this paper presents a three-dimensional (3D) object registration solution in the view-based framework, which incorporates a new image feature named as chain-of-lines feature (COLF). The COLF, constructed by several directed line segments with geometric constraints, enhances the robustness of the registration by establishing multiple correspondences between the two-dimensional (2D) image and 3D model simultaneously. In order to save on computational cost and improve the efficiency of the COLF-based registration, a compression algorithm based on the local breadth-first search (LBFS) strategy is also proposed; this adaptively reduces the number of keyframes according to the target model. In the experimental evaluation, the combination of COLF and LBFS-based keyframe compression shows higher registration success rate and faster speed in comparison with conventional registration methods. Moreover, an AR assembly guidance prototype system for reductors is introduced to illustrate the application of the proposed registration method.     

Exploring legibility of augmented reality X-ray

Virtual objects can be visualized inside real objects using augmented reality (AR). This visualization is called AR X-ray because it gives the impression of seeing through the real object. In standard AR, virtual information is overlaid on top of the real world. To position a virtual object inside an object, AR X-ray requires partially occluding the virtual object with visually important regions of the real object. In effect, the virtual object becomes less legible compared to when it is completely unoccluded. Legibility is an important consideration for various applications of AR X-ray. In this research, we explored legibility in two implementations of AR X-ray, namely, edge-based and saliency-based. In our first experiment, we explored on the tolerable amounts of occlusion to comfortably distinguish small virtual objects. In our second experiment, we compared edge-based and saliency-based AR X-ray methods when visualizing virtual objects inside various real objects. Moreover, we benchmarked the legibility of these two methods against alpha blending. From our experiments, we observed that users have varied preferences for proper amounts of occlusion cues for both methods. The partial occlusions generated by the edge-based and saliency-based methods need to be adjusted depending on the lighting condition and the texture complexity of the occluding object. In most cases, users identify objects faster with saliency-based AR X-ray than with edge-based AR X-ray. Insights from this research can be directly applied to the development of AR X-ray applications.     

基于增强现实技术的景观规划系统

增强现实是把计算机产生的虚拟物体或其他信息合成到用户看到的真实世界中的一种技术，利用增强现实技术，设计人员能够有效地进行建筑布局的规划和设计，本文介绍了一个典型的增强现实系统-景观规划系统的实现，针对合成虚拟景物和真实场景过程中需要解决的几何一致性和光照一致性问题，采用了局部三维重建和交互的相机定标技术来实现几何一致性；对于光照一致性，则采用了光照交互指定和自动恢复相结合的方法，同时还考虑了虚拟物体投射的阴影，针对不同场景的实际应用，依据系统生成的结果合成图象，设计人员能够比较有效地完成对景观规划结果的直观评估。     

增强现实系统中大范围磁跟踪注册问题研究

为提高增强现实系统在大范围工作空间下的三维注册精度,分析空间网格线性插值法产生误差的原因,结合反距离权重插值算法,提出一种对磁跟踪校正的融合算法。改进原搜索策略以及插值算法在大范围工作空间下的不足,探讨双发射器协同工作的可能性,并设计解决方案。实验结果表明,双发射器的跟踪注册方案能扩大增强现实系统的工作空间和交互范围。     

Webizing mobile augmented reality content

This paper presents a content structure for building mobile augmented reality (AR) applications in HTML5 to achieve a clean separation of the mobile AR content and the application logic for scaling as on the Web. We propose that the content structure contains the physical world as well as virtual assets for mobile AR applications as document object model (DOM) elements and that their behaviour and user interactions are controlled through DOM events by representing objects and places with a uniform resource identifier. Our content structure enables mobile AR applications to be seamlessly developed as normal HTML documents under the current Web eco-system.     

The Role of 3-D Sound in Human Reaction and Performance in Augmented Reality Environments

Three-dimensional sound's effectiveness in virtual reality (VR) environments has been widely studied. However, due to the big differences between VR and augmented reality (AR) systems in registration, calibration, perceptual difference of immersiveness, navigation, and localization, it is important to develop new approaches to seamlessly register virtual 3-D sound in AR environments and conduct studies on 3-D sound's effectiveness in AR context. In this paper, we design two experimental AR environments to study the effectiveness of 3-D sound both quantitatively and qualitatively. Two different tracking methods are applied to retrieve the 3-D position of virtual sound sources in each experiment. We examine the impacts of 3-D sound on improving depth perception and shortening task completion time. We also investigate its impacts on immersive and realistic perception, different spatial objects identification, and subjective feeling of "human presence and collaboration". Our studies show that applying 3-D sound is an effective way to complement visual AR environments. It helps depth perception and task performance, and facilitates collaborations between users. Moreover, it enables a more realistic environment and more immersive feeling of being inside the AR environment by both visual and auditory means. In order to make full use of the intensity cues provided by 3-D sound, a process to scale the intensity difference of 3-D sound at different depths is designed to cater small AR environments. The user study results show that the scaled 3-D sound significantly increases the accuracy of depth judgments and shortens the searching task completion time. This method provides a necessary foundation for implementing 3-D sound in small AR environments. Our user study results also show that this process does not degrade the intuitiveness and realism of an augmented audio reality environment