Rendering techniques for mixed reality

In mixed reality (MR) design review, the aesthetics of a virtual prototype is assessed by integrating a virtual model into a real-world environment and inspecting the interaction between the model and the environment (lighting, shadows and reflections) from different points of view. The visualization of the virtual model has to be as realistically as possible to provide a solid basis for this assessment and interactive rendering speed is mandatory to allow the designer to examine the scene from arbitrary positions. In this article we present a real-time rendering engine specifically tailored to the needs of MR visualization. The renderer utilizes pre-computed radiance transfer to calculate dynamic soft-shadows, high dynamic range images and image-based lighting to capture incident real-world lighting, approximate bidirectional texture functions to render materials with self-shadowing, and frame post-processing filters (bloom filter and an adaptive tone mapping operator). The proposed combination of rendering techniques provides a trade-off between rendering quality and required computing resources which enables high quality rendering in mobile MR scenarios. The resulting image fidelity is superior to radiosity-based techniques because glossy materials and dynamic environment lighting with soft-shadows are supported. Ray tracing-based techniques provide higher quality images than the proposed system, but they require a cluster of computers to achieve interactive frame rates which prevents these techniques from being used in mobile MR (especially outdoor) scenarios. The renderer was developed in the European research project IMPROVE (FP6-IST-2-004785) and is currently extended in the MAXIMUS project (FP7-ICT-1-217039) where hybrid rendering techniques which fuse PRT and ray tracing are developed.     

Design for emergence: experiments with a mixed reality urban playground game

In this paper we present our work in the design of ubiquitous social experiences, aiming to foster group participation and spontaneous playful behaviours in a city environment. We outline our approach of design for emergence: to provide just enough of a game context and challenge for people to be creative, to extend and enrich the experience of play through their interaction in the real world. CitiTag is our mixed reality testbed, a wireless location-based multiplayer game based on the concept of playground ‘tag’. We describe the design and implementation of CitiTag and discuss results from two user studies.     

Virtual Bounds: a teleoperated mixed reality

This paper introduces a mixed reality workspace that allows users to combine physical and computer-generated artifacts, and to control and simulate them within one fused world. All interactions are captured, monitored, modeled and represented with pseudo-real world physics. The objective of the presented research is to create a novel system in which the virtual and physical world would have a symbiotic relationship. In this type of system, virtual objects can impose forces on the physical world and physical world objects can impose forces on the virtual world. Virtual Bounds is an exploratory study allowing a physical probe to navigate a virtual world while observing constraints, forces, and interactions from both worlds. This scenario provides the user with the ability to create a virtual environment and to learn to operate real-life probes through its virtual terrain.     

Construction of a computer-simulated mixed reality environment for virtual factory layout planning

To survive the cut-throat competition in the manufacturing industry, many companies have introduced digital manufacturing technology. Digital manufacturing technology not only shortens the product development cycle times but also improves the precision of engineering simulation. However, building the virtual objects needed for a digital manufacturing environment requires skilled human resources; it is also costly and time-consuming. A high precision environment with the similar resources is also needed for a high precision simulation. In this paper, we propose a method of constructing a mixed reality-based digital manufacturing environment. The method integrates real objects, such as real images, with the virtual objects of a virtual manufacturing system. This type of integration minimizes the cost of implementing virtual objects and enhances the user's sense of reality. We studied several methods and derived a general framework for the system. Finally, we developed our idea into a virtual factory layout planning system. To assign the pose and position of real objects in virtual space, we applied a circle-based tracking method which uses a safety sign instead of the planar-square-shaped marker generally used for registration. Furthermore, we developed the framework to encapsulate simulation data from legacy data and process data for visualization based on mixed reality.     

The feasibility of a mixed reality surgical training environment

The Sheffield knee arthroscopy training system (SKATS) was originally a visual-based virtual environment without haptic feedback, but has been further developed as a mixed reality-training environment through the use of tactile augmentation (or passive haptics). The design of the new system is outlined and then tested. In the first experiment described, the effect of tactile augmentation on performance is considered by comparing novice performance using the original and mixed reality system. In the second experiment the mixed reality system is assessed in terms of construct validity by comparing the performance of users with differing levels of surgical expertise. The results are discussed in terms of the validity of a mixed reality environment for training knee arthroscopy.     

DVV: a taxonomy for mixed reality visualization in image guided surgery

Mixed reality visualizations are increasingly studied for use in image guided surgery (IGS) systems, yet few mixed reality systems have been introduced for daily use into the operating room (OR). This may be the result of several factors: the systems are developed from a technical perspective, are rarely evaluated in the field, and/or lack consideration of the end user and the constraints of the OR. We introduce the Data, Visualization processing, View (DVV) taxonomy which defines each of the major components required to implement a mixed reality IGS system. We propose that these components be considered and used as validation criteria for introducing a mixed reality IGS system into the OR. A taxonomy of IGS visualization systems is a step toward developing a common language that will help developers and end users discuss and understand the constituents of a mixed reality visualization system, facilitating a greater presence of future systems in the OR. We evaluate the DVV taxonomy based on its goodness of fit and completeness. We demonstrate the utility of the DVV taxonomy by classifying 17 state-of-the-art research papers in the domain of mixed reality visualization IGS systems. Our classification shows that few IGS visualization systems' components have been validated and even fewer are evaluated.     

A method of creating a transparent space based on mixed reality

In this article, we propose a novel idea to create a transparent space. Accidents often happen when a person, a motorcycle, a car, etc., comes out suddenly from a blind corner. In such a place, if the shielding objects become transparent, hidden objects reveal themselves. This may contribute to preventing a lot of accidents. The proposed method creates a transparent space by the employment of a mixed-reality technique and a simple three-dimensional (3-D) recovery technique. The 3-D virtual space is created by a single camera image in real-time, and it is superposed onto a real space employing see-through vision. The observer can watch a dynamic scene of the invisible space through the object in real-time. In the experiment performed, an observer could watch persons walking around in a space hidden by an opaque partition. Future application of this technique may include its installation at various corners in buildings or at intersections on roads with bad visibility. This would surely contribute to reducing traffic accidents in future. This work was presented in part at the 11th International Symposium on Artificial Life and Robotics, Oita, Japan, January 23–25, 2006     

增强现实系统软件平台的设计与实现

在对于一个增强现实系统的整体结构与各组成模块进行完整分析的基础之上,使用OpenCV、Coin3D和ARToolKit这3个软件开发包在Visual Studio 2003.net环境下开发了一个增强现实系统的软件平台.主要阐述了虚实环境的构建、三维注册、标志物检测、图像处理,视频融合等主要功能模块的实现原理与方法,最后给出了系统实际运行的效果.     

Virtual reality software package for implementing motor learning and rehabilitation experiments

Virtual reality games for rehabilitation are attracting increasing growth. In particular, there is a demand for games that allow therapists to identify an individual’s difficulties and customize the control of variables, such as speed, size, distance, as well as visual and auditory feedback. This study presents and describes a virtual reality software package (Bridge Games) to promote rehabilitation of individuals living with disabilities and highlights preliminary researches of its use for implementing motor learning and rehabilitation. First, the study presents seven games in the software package that can be chosen by the rehabilitation team, considering the patient’s needs. All game characteristics are described including name, function presentation, objective and valuable measurements for rehabilitation. Second, preliminary results illustrate some applications of two games, considering 343 people with various disabilities and health status. Based on the results, in the Coincident Timing game, there was a main effect of movement sensor type (in this instance the most functional device was the keyboard when compared with Kinect and touch screen) on average time reached by sample analyzed, F(2, 225) = 4.42, p < 0.05. Similarly, in the Challenge! game, a main effect was found for movement sensor type. However, in this case, touch screen provided better performance than Kinect and Leap Motion, F(2, 709) = 5.90, p < 0.01. Thus, Bridge Games is a possible software game to quantify motor learning. Moreover, the findings suggest that motor skills might be practiced differently depending on the environmental interface in which the game may be used.     

Heads up and camera down: a vision-based tracking modality for mobile mixed reality

Anywhere Augmentation pursues the goal of lowering the initial investment of time and money necessary to participate in mixed reality work, bridging the gap between researchers in the field and regular computer users. Our paper contributes to this goal by introducing the GroundCam, a cheap tracking modality with no significant setup necessary. By itself, the GroundCam provides high frequency, high resolution relative position information similar to an inertial navigation system, but with significantly less drift. We present the design and implementation of the GroundCam, analyze the impact of several design and run-time factors on tracking accuracy, and consider the implications of extending our GroundCam to different hardware configurations. Motivated by the performance analysis, we developed a hybrid tracker that couples the GroundCam with a wide area tracking modality via a complementary Kalman filter, resulting in a powerful base for indoor and outdoor mobile mixed reality work. To conclude, the performance of the hybrid tracker and its utility within mixed reality applications is discussed.     

基于多种语言混合编程的优化软件设计

针对爆炸结构的设计及优化,采用APDL、Matlab和VC 语言混合编程的方法优化软件的设计,不但可以方便地实现优化算法和设计友好的用户界面,还能实现数值模拟计算程序与优化程序联合使用,可以有效地缩短程序设计时间.对LS-DYNA分析计算结果的整理及结构材料信息文件的修改给出了解决办法,实现了全自动优化过程,提高了优化效率.优化软件的设计实例表明了该方法的可行性和有效性.     

A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality

Recent user interface concepts, such as multimedia, multimodal, wearable, ubiquitous, tangible, or augmented-reality-based (AR) interfaces, each cover different approaches that are all needed to support complex human–computer interaction. Increasingly, an overarching approach towards building what we call ubiquitous augmented reality (UAR) user interfaces that include all of the just mentioned concepts will be required. To this end, we present a user interface architecture that can form a sound basis for combining several of these concepts into complex systems. We explain in this paper the fundamentals of DWARFs user interface framework (DWARF standing for distributed wearable augmented reality framework) and an implementation of this architecture. Finally, we present several examples that show how the framework can form the basis of prototypical applications.     

Reliable behaviour simulation of product interface in mixed reality

The validation of a product interface is often a critical issue in the design process. Virtual reality and mixed reality (MR) are able to enhance the interactive simulation of the product human–machine interface (HMI), as these technologies allow engineers to directly involve end users in the usability assessment. This paper describes a MR environment specifically addressed to the usability evaluation of a product interface, which allows the simulation of the HMI behaviour using the same models and the same software employed by engineers during the design phase. Our approach is based on the run-time connection between the visualisation software and the simulators used for product design and analysis. In particular, we use Matlab/Simulink to model and simulate the product behaviour, and Virtools to create the interactive MR environment in which the end user can test the product. Thanks to this architecture, any modification done on the behaviour models is immediately testable in MR.     

HoloFlows: modelling of processes for the Internet of Things in mixed reality

Software and Systems Modeling - Our everyday lives are increasingly pervaded by digital assistants and smart devices forming the Internet of Things (IoT). While user interfaces to directly monitor...     

Earth science learning in SMALLab: A design experiment for mixed reality

Conversational technologies such as email, chat rooms, and blogs have made the transition from novel communication technologies to powerful tools for learning. Currently virtual worlds are undergoing the same transition. We argue that the next wave of innovation is at the level of the computer interface, and that mixed-reality environments offer important advantages over prior technologies. Thus, mixed reality is positioned to have a broad impact on the future of K-12 collaborative learning. We propose three design imperatives that arise from our ongoing work in this area grounded in research from the learning sciences and human-computer interaction. By way of example, we present one such platform, the Situated Multimedia Arts Learning Lab [SMALLab]. SMALLab is a mixed-reality environment that affords face-to-face interaction by colocated participants within a mediated space. We present a recent design experiment that involved the development of a new SMALLab learning scenario and a collaborative student participation framework for a 3-day intervention for 72 high school earth science students. We analyzed student and teacher exchanges from classroom sessions both during the intervention and during regular classroom instruction and found significant increases in the number of student-driven exchanges within SMALLab. We also found that students made significant achievement gains. We conclude that mixed reality can have a positive impact on collaborative learning and that it is poised for broad dissemination into mainstream K-12 contexts.     

基于混合现实的矿用设备维修指导系统

针对矿用设备现场维修知识和专业指导不足导致维修效率低、周期长,虚拟现实和增强现实技术实现平台一般为头戴式设备、手机等,设备的移动性很好,但用户只能依赖交互手柄、触摸屏进行交互,视觉体验缺乏真实性,设计了一种基于混合现实的矿用设备维修指导系统。该系统基于Unity3D虚拟现实平台,采用混合现实设备HoloLens实时显示维修指导方案,实现了矿用设备原理虚拟仿真、基于二维码的设备管理、设备维修指导和远程专家协作等功能。HoloLens通过虚拟文字、虚拟设备模型、动画及语音提示等方式将维修指导信息直观地呈现给维修人员,维修人员可通过视线跟踪、语音交互、手势识别等方式与HoloLens交互。采煤机齿轮磨损故障维修指导应用实例验证了该系统的实用性和有效性。     

General-purpose modular hardware and software framework for mobile outdoor augmented reality applications in engineering

This paper presents a reusable, general-purpose, mobile augmented reality (AR) framework developed to address the critical and repetitive challenges specific to visualization in outdoor AR. In all engineering applications of AR developed thus far, basic functionality that supports accurate user registration, maximizes the range of user motion, and enables data input and output has had to be repeatedly re-implemented. This is primarily due to the fact that designed methods have been traditionally custom created for their respective applications and are not generic enough to be readily shared and reused by others. The objective of this research was to remedy this situation by designing and implementing a reusable and pluggable hardware and software framework that can be used in any AR application without the need to re-implement low-level communication interfaces with selected hardware. The underlying methods of hardware communication as well as the object-oriented design (OOD) of the reusable interface are presented. Details on the validation of framework reusability and pluggability are also described.