Future multimedia user interfaces

No matter how powerful or elegant the technologies underlying multimedia computing, it is the user interface that ultimately determines how these systems will be used. We provide an overview of two emerging areas of user-interface research that will play major roles in future multimedia systems: virtual environments and ubiquitous computing. We discuss what they are and why they are important, and describe their relationship to current multimedia systems. While mouse and window-based user interfaces provide access to 2D graphics, video, and sound, virtual environments not only involve interactive 3D graphics, but further address all our senses through the use of spatial audio, haptics, and other modalities. While it is still a research goal to provide high-throughput video on demand, ubiquitous computing will require that multimedia network infrastructure also support extremely low-latency interaction among large numbers of mobile wireless users. To arrive at an understanding of the needs, capabilities, and potential of these new paradigms, we introduce their terminology and technology, and explain the basic system architectures being explored, leading up to a discussion of key current research issues and future directions.     

Real-time 3D motion capture by monocular vision and virtual rendering

Networked 3D virtual environments allow multiple users to interact over the Internet by means of avatars and to get some feeling of a virtual telepresence. However, avatar control may be tedious. 3D sensors for motion capture systems based on 3D sensors have reached the consumer market, but webcams remain more widespread and cheaper. This work aims at animating a user’s avatar by real-time motion capture using a personal computer and a plain webcam. In a classical model-based approach, we register a 3D articulated upper-body model onto video sequences and propose a number of heuristics to accelerate particle filtering while robustly tracking user motion. Describing the body pose using wrists 3D positions rather than joint angles allows efficient handling of depth ambiguities for probabilistic tracking. We demonstrate experimentally the robustness of our 3D body tracking by real-time monocular vision, even in the case of partial occlusions and motion in the depth direction.     

A live video streaming system for intuitive human-system interaction

In this paper, we propose an intuitive live video streaming system based on virtual reality technologies among people who are far apart. This system is a kind of server-client system, and can provide remote users with virtual 3D audiovisual fields in real time via a very-high-speed network. The server captures audio and video data from its clients, compiles them into a 3D audiovisual scene at a virtual conference, and broadcasts it to the clients. At the present stage, our system captures 2 videos and creates one 3D video at a time. Our system can play 3D audiovisual contents on Windows XP systems as well as on CAVE systems. Currently, our system can play the 3D video contents at about 2.36 fps under a LAN environment. This work was presented in part at the 12th International Symposium on Artificial Life and Robotics, Oita, Japan, January 25–27, 2007     

A virtual environment-based system for the navigation of underwater robots

Efficient teleoperation of underwater robot requires clear 3D visual information of the robot's spatial location and its surrounding environment. However, the performance of existing telepresence systems is far from satisfactory. In this paper, we present our virtual telepresence system for assisting tele-operation of an underwater robot. This virtual environment-based telepresence system transforms robot sensor data into 3D synthetic visual information of the workplace based on its geometrical model. It provides the operators with a full perception of the robot's spatial location. In addition, we propose a robot safety domain to overcome the robot's location offset in the virtual environment caused by its sensor errors. The software design of the system and how a safety domain can be used to overcome robot location offset in virtual environment will be examined. Experimental tests and its result analysis will also be presented in this paper.     

A voyage into virtual reality. Networking our VR lab to Iowa middleschools and high schools

The Iowa Communications Network (ICN) is a statewide, state administered, fiber optics network. Iowa State University (ISU) is one of the fiber optics endpoints and one of the main entities using the ICN to reach the Iowa community. As part of the ICN outreach activities at ISU, we are investigating how the ICN can be integrated with our research projects in virtual reality (VR). We're particularly interested in how the ICN can enable remote collaborations in immersive virtual environments with sites that don't have any or have limited local resources for VR. We present our first project with the ICN to bring our VR lab to remote middle schools and high schools in Iowa. This project was done in collaboration with Iowa Public Television (IPTV), another ICN endpoint. We connected to 86 schools simultaneously and reached about 2000 students. Ten sites were in interactive two-way video and audio mode. The rest were in viewing-only video and audio mode. We discuss the motivation, technical details, results, and future directions     

Towards video-based immersive environments

Video provides a comprehensive visual record of environment activity over time. Thus, video data is an attractive source of information for the creation of virtual worlds which require some real-world fidelity. This paper describes the use of multiple streams of video data for the creation of immersive virtual environments. We outline our multiple perspective interactive video (MPI-Video) architecture which provides the infrastructure for the processing and analysis of multiple streams of video data. Our MPI-Video system performs automated analysis of the raw video and constructs a model of the environment and object activity within this environment. This model provides a comprehensive representation of the world monitored by the cameras which, in turn, can be used in the construction of a virtual world. In addition, using the information produced and maintained by the MPI-Video system, our immersive video system generates virtual video sequences. These are sequences of the dynamic environment from an arbitrary view point generated using the real camera data. Such sequences allow a user to navigate through the environment and provide a sense of immersion in the scene. We discuss results from our MPI-Video prototype, outline algorithms for the construction of virtual views and provide examples of a variety of such immersive video sequences.     

虚拟现实力觉临场感遥控作业系统的研究进展

本文阐述了虚拟现实技术在力觉临场感遥控作业系统中的研究和应用背景、状况和主要内容,综述了虚拟现实力觉临场遥控作业实系统的国内外最近研究进展,并介绍了目前研究亟需解决的关键技术,指出虚拟现实力觉临场感遥控作业系统的研究对临场感技术以至于非确定性环境下作业的第三代机器人的发展有着非常重要的理论意义和应用价值。     

利用高级在轨系统实现高速同/异步混合复接

空间数据传输系统中,音、视频数据的传输变得越来越重要.由于音、视频传输对等时性和实时性要求高,并且视频数据的速率高,数据量大,传统的做法是为音、视频信息单独开辟一个信道传输,但这样会耗费大量的资源.而将视频、音频和异步数据混合复接传输,可以提高信道利用率.针对这种任务,利用国际空间数据系统咨询委员会（CCSDS）高级在轨系统（AOS）建议,提出了两级复用的方案和虚拟信道调度的算法,以满足视频、音频数据传输的等时性和异步数据传输的灵活性,并设计了演示系统.     

Scalable and Efficient Video Coding Using 3-D Modeling

In this paper, we present a three-dimensional (3D) model-based video coding scheme for streaming static scene video in a compact way but also enabling time and spatial scalability according to network or terminal capability and providing 3D functionalities. The proposed format is based on encoding the sequence of reconstructed models using second-generation wavelets, and efficiently multiplexing the resulting geometric, topological, texture, and camera motion binary representations. The wavelets decomposition can be adaptive in order to fit to images and scene contents. To ensure time scalability, this representation is based on a common connectivity for all 3D models, which also allows straightforward morphing between successive models ensuring visual continuity at no additional cost. The method proves to be better than previous methods for video encoding of static scenes, even better than state-of-the-art video coders such as H264 (also known as MPEG AVC). Another application of our approach are smoothing camera path for suppression of jitter from hand-held acquisition and the fast transmission and real-time visualization of virtual environments obtained by video capture, for virtual or augmented reality and interactive walk-through in photo-realistic 3D environments around the original camera path     

多媒体视、音频信号处理系统的集成与优化

在多媒体视、音频信号处理系统中,主要涉及数据通信、音频处理,复合视频信号（ＰＡＬ、ＮＴＳＣ）的变换、合成视频信号（Ｒ、Ｇ、Ｂ）的传输４大模块。讨论了如何结合新技术和新器件来实现各模块的高度集成和优化设计,从而使系统在性能指标、功能、结构体系等方面体现出良好的品质。     

基于JSP的网络视音频播放系统设计与实现

根据流媒体传输原理,在局域网的基础上模拟基于Web的视频和音频播放系统,实现用户信息管理、听音频、看视频、文件的添加、删除、修改、上传及搜索功能等,从而设计出符合现在人们需求的视音频播放系统,为网络时代的人们提供方便、快捷的视音频点播节目,提供更加人性化设置。     

Transitioning Desktops to Set Tops

Video provides the greatest realism in multimedia and also demands the most from computer systems. Innovations already enable conventional desktops, such as RISC workstations, to handle compressed audio and low-resolution video signals. Similarly, interactive, real-time, 3D animation graphics produce data processing and transmission challenges. Meeting these challenges will transition desktop systems to televisions with set-top boxes, linking broadcast quality video and audio to the interactivity of a computer     

Data generation and representation method for 3D video conferencing using programming by demonstration

Video conferencing provides an environment for multiple users linked on a network to have meetings. Since a large quantity of audio and video data are transferred to multiple users in real time, research into reducing the quantity of data to be transferred has been drawing attention. Such methods extract and transfer only the features of a user from video data and then reconstruct a video conference using virtual humans. The disadvantage of such an approach is that only the positions and features of hands and heads are extracted and reconstructed, whilst the other virtual body parts do not follow the user. In order to enable a virtual human to accurately mimic the entire body of the user in a 3D virtual conference, we examined what features should be extracted to express a user more clearly and how they can be reproduced by a virtual human. This 3D video conferencing estimates the user’s pose by comparing predefined images with a photographed user’s image and generates a virtual human that takes the estimated pose. However, this requires predefining a diverse set of images for pose estimation and, moreover, it is difficult to define behaviors that can express poses correctly. This paper proposes a framework to automatically generate the pose-images used to estimate a user’s pose and the behaviors required to present a user using a virtual human in a 3D video conference. The method for applying this framework to a 3D video conference on the basis of the automatically generated data is also described. In the experiment, the framework proposed in this paper was implemented in a mobile device. The generation process of poses and behaviors of virtual human was verified. Finally, by applying programming by demonstration, we developed a system that can automatically collect the various data necessary for a video conference directly without any prior knowledge of the video conference system.     

一种视频传输处理系统的综合解决方案

详细讨论并给出了一种适合于实际应用的视频传输系统的设计原理和实现方法。设计中为提高系统资源的使用率、网络带宽的应用率,提出了网络通信变VBR为动态CBR的方法,通过反馈机制将信道、传输、接收效果与编码直接相联系,并采用编解码结合、优先级控制和虚拟信道传输的方式,达到全系统自适应控制。     

基于ARM的无线音视频监控系统设计

介绍了一种基于ARM硬件平台和WinCE软件平台的无线音视频监控系统设计方案.该设计应用三星公司的S3C2440A为主芯片,结合摄像头、麦克风和GPRS模块等实现数据的采集、处理和网络传送.S3C2440A具有丰富的硬件接口资源,采用H.264作为音视频数据压缩的核心算法,具有很高的数据压缩比率,适合无线传输.     

Virtual Video Prototyping

Computing power is an integrated part of our physical environment, and since our physical environment is three-dimensional, the virtual studio technology, with its unique potential for visualizing digital 3D objects and environments along with physical objects, offers an obvious path to pursue in order to envision future usage scenarios in the domain of pervasive computing.We label the work method virtual video prototyping, which grew out of a number of information systems design techniques along with approaches to visualization in the field of architecture and set design.We present a collection of virtual video prototyping cases and use them as the platform for a discussion, which pinpoint advantages and disadvantages of working with virtual video prototyping as a tool for communication, experimentation and reflection in the design process. Based on more than ten cases we have made the observations that virtual video prototypes 1) are a powerful medium of communication in development teams and for communication with industry partners and potential investors, 2) support both testing and generating ideas 3) are particular suited for addressing spatial issues and new ways of interacting. In addition practical use of virtual video prototypes has indicated the need to take into account some critical issues including a) production resources, b) hand-on experience, and c) the seductive power of virtual video prototypes.     

Metaphor and Manifestation Cross-Reality with Ubiquitous Sensor/Actuator Networks

A series of projects by the MIT Media Lab's Responsive Environments Group explore ways of bridging the rapidly expanding reach of networked electronic sensors with the limited realm of human perception. These include various implementations of cross-reality which render and manifest phenomena between the real world and shared online virtual environments via densely embedded sensor and actuator networks. We visualize information from ubiquitously deployed real-world smart power strips and sensor-rich media portals at different levels of abstraction through analogous Second Life constructs. Conversely, we manifest virtual world events into physical space using on-platform actuators and displays. We also show a set of simpler 2D visualizations that enable mobile devices to efficiently browse and interact with sensor network data. We touch on a recently developed system that uses a small badge to passively manage dynamic privacy in environments such as these that stream potentially revealing information across the real/virtual divide. These technologies' application areas involve fluid browsing of and interaction with the geographically dispersed real world in an unconstrained virtual environment and ubiquitous multiscale telepresence.     

Virtual reality movies-real-time streaming of 3D objects

Powerful servers for computation and storage, high-speed networking resources, and high-performance 3D graphics workstations, which are typically available in scientific research environments, potentially allow the development and productive application of advanced distributed high-quality multimedia concepts. Several bottlenecks, often caused by the inefficient design and software implementation of current systems, prevent utilization of the offered performance of existing hardware and networking resources. We present a system architecture, which supports streamed online presentation of series of 3D objects. In the case of expensive simulations on a supercomputer, results are increasingly represented as 3D geometry to support immersive exploration, presentation, and collaboration techniques. Accurate representation and high-quality display are fundamental requirements to avoid misinterpretation of the data. Our system consists of the following parts: a preprocessor to create a special 3D representation – optimized under transmission and streamed presentation issues in a high-performance working environment, an efficiently implemented streaming server, and a client. The client was implemented as a web browser plugin, integrating a viewer with high-quality virtual reality presentation (stereoscopic displays), interaction (tracking devices), and hyperlinking capabilities.     

Low-bandwidth 3D visual telepresence system

We present a methodology to develop a low-cost, low-bandwidth visual telepresence system using commodity depth sensors. To obtain a precise representation of the participants, we fuse together multiple views extracted using a deep background subtraction method. We build a proof-of-concept display composed of a video projector and a quadrangular pyramid made of acrylic, to demonstrate the visualization of a remote person without the need for head-mounted displays. Our system represents an attempt to democratize high-fidelity 3D telepresence using off-the-shelf components.

     

Encoding Video Narration as Text

This paper describes a solution to the problem of transmitting “talking head” video in real-time across low-bandwidth transmission media. Our solution is based on a video reconstruction system, which can generate realistic audio-visual narrations from arbitrary pieces of text. The system uses standard synthetic speech techniques to create an audio track and then produces a synchronized “talking head” using full-frame morphing of real-world facial images (key-frames). We discuss issues of original data capture, transmission protocols, synchronization, interpolation, and the modeling of various physical activities which are vital to maintaining the “plausibility” of the final synthetic video sequence.