Motion capture and human pose reconstruction from a single-view video sequence

We propose a framework to reconstruct the 3D pose of a human for animation from a sequence of single-view video frames. The framework for pose construction starts with background estimation and the performer?s silhouette is extracted using image subtraction for each frame. Then the body silhouettes are automatically labeled using a model-based approach. Finally, the 3D pose is constructed from the labeled human silhouette by assuming orthographic projection. The proposed approach does not require camera calibration. It assumes that the input video has a static background, it has no significant perspective effects, and the performer is in an upright position. The proposed approach requires minimal user interaction.     

Points‐based user interface for character posing

We present a points‐based user interface for character posing. In our method, users insert a number of three‐dimensional (3D) points in a virtual environment. The system performs a linear search of a motion capture database for the best matched pose and then places the pose immediately in the virtual environment to be overlapped with the input points. For a fast and precise distance computation between the input points and the example poses from the database, we developed a closed‐form solution of the 3D points registration problem. To demonstrate the easiness and usability of our approach, we built a motion database including various kinds of human motion and conducted a user study of character posing tasks with non‐expert users. Copyright © 2016 John Wiley & Sons, Ltd.     

Vision-Based Hand Interaction in Augmented Reality Environment

A new vision-based framework and system for human–computer interaction in an Augmented Reality environment is presented in this article. The system allows the users to interact with computer-generated virtual objects using their hands directly. With an efficient color segmentation algorithm, the system is adaptable to different light conditions and backgrounds. It is also suitable for real-time applications. The dominant features on the palm are detected and tracked to estimate the camera pose. After the camera pose relative to the user's hand has been reconstructed, 3D virtual objects can be augmented naturally onto the palm for the user to inspect and manipulate. With off-the-shelf web camera and computer, natural bare-hand based interactions with 2D and 3D virtual objects can be achieved with low cost.     

Skin colour segmentation based 2D and 3D human pose modelling using Discrete Wavelet Transform

Tracking of human body and their pose models from a video data is important in the indoor surveillance applications. The objective of this paper is to develop a human pose modelling in two-dimensional and three dimensional views using wavelet based skin colour segmentation for the monocular videos. The approximation information of the Discrete Wavelet Transform (DWT) is used for segmenting the human body from the video frame and to find the skin features. The human pose modelling is proposed by detecting thirteen features from the upper body as well as in the lower body. Here, twelve human poses are considered for the analysis. In this paper, the hardware system is implemented for monitoring the human poses by sending the SMS message. Also, the efficiencies of 2D modelling and 3D modelling for the pre-defined poses are evaluated.     

Real-time 3D motion capture by monocular vision and virtual rendering

Networked 3D virtual environments allow multiple users to interact over the Internet by means of avatars and to get some feeling of a virtual telepresence. However, avatar control may be tedious. 3D sensors for motion capture systems based on 3D sensors have reached the consumer market, but webcams remain more widespread and cheaper. This work aims at animating a user’s avatar by real-time motion capture using a personal computer and a plain webcam. In a classical model-based approach, we register a 3D articulated upper-body model onto video sequences and propose a number of heuristics to accelerate particle filtering while robustly tracking user motion. Describing the body pose using wrists 3D positions rather than joint angles allows efficient handling of depth ambiguities for probabilistic tracking. We demonstrate experimentally the robustness of our 3D body tracking by real-time monocular vision, even in the case of partial occlusions and motion in the depth direction.     

A fast and stable facial interface using integral projections

Integral projections are a useful technique in many computer vision problems. In this paper, we present a perceptual interface which allows us to navigate through a virtual 3D world by using the movements of the face of human users. The system applies advanced computer vision techniques to detect, track, and estimate the pose of the user’s head. The core of the proposed approach consists of a face tracker, which is based on the computation, alignment, and analysis of integral projections. This technique provides a robust, accurate, and stable 2D location of the face in each frame of the input video. Then, 3D location and orientation of the head are estimated, using some predefined heuristics. Finally, the resulting 3D pose is transformed into control signals for the navigation in the virtual world. The proposed approach has been implemented and tested in a prototype, which is publicly available. Some experimental results are shown, proving the feasibility of the method. The perceptual interface is fast, stable, and robust to facial expression and illumination conditions.     

Spatial conferencing using a wearable computer

Wearable computers provide constant access to computing and communications resources; however, there are many unanswered questions as to how this computing power can be used to enhance communication. We describe a wearable augmented reality communication space that uses spatialised 3D graphics and audio cues to aid communication. The user is surrounded by virtual avatars of the remote collaborators that they can interact with using natural head and body motions. The use of spatial cues means that the conferencing space can potentially support dozens of simultaneous users. We report on two experiments that show users can understand speakers better with spatial rather than non-spatial audio, and that minimal visual cues may be sufficient to distinguish between speakers. Additional informal user studies with real conference participants suggest that wearable communication spaces may offer significant advantages over traditional communication devices.     

Application of Multicast-based Video Conference on CERNET Backbone

Multicast-based video conference is a representative application in advanced network. In multi-point video conference using multicast can get better efficiency facilitated by inner-group broadcast mechanism. In the application, the multicast-based network resources assignment, management and security should be considered together. This paper presents a framework model of multicast-based video conferencing application with three layers. And a practical multicast-based video conferencing is implemented in CERNET(China Education and Research Network) backbone. The practice is valuable for the development of multicast-based video conferencing application in China.     

Coupled Action Recognition and Pose Estimation from Multiple Views

Action recognition and pose estimation are two closely related topics in understanding human body movements; information from one task can be leveraged to assist the other, yet the two are often treated separately. We present here a framework for coupled action recognition and pose estimation by formulating pose estimation as an optimization over a set of action-specific manifolds. The framework allows for integration of a 2D appearance-based action recognition system as a prior for 3D pose estimation and for refinement of the action labels using relational pose features based on the extracted 3D poses. Our experiments show that our pose estimation system is able to estimate body poses with high degrees of freedom using very few particles and can achieve state-of-the-art results on the HumanEva-II benchmark. We also thoroughly investigate the impact of pose estimation and action recognition accuracy on each other on the challenging TUM kitchen dataset. We demonstrate not only the feasibility of using extracted 3D poses for action recognition, but also improved performance in comparison to action recognition using low-level appearance features.     

结合个性化建模和深度数据的三维人体姿态估计

利用深度传感器估计三维人体姿态是计算机视觉领域的一个重要问题,在人机交互、虚拟现实和动画设计等领域有重要的应用价值.针对该问题的主流方法是自底向上的方法,这类方法一般采用分类、回归或检索技术,可以直接从深度数据中估计三维肢体姿态,在人机交互中得到了很广泛的应用.但是这类方法依赖于大规模的姿态数据库,而且结果不够精确.本文提出一种结合个性化人体建模和深度数据的三维姿态估计方法,首先对运动对象建立三维虚拟人模型,然后利用该个性化的虚拟人模型与深度数据之间的点匹配关系构造姿态优化的目标函数,通过迭代优化目标函数,估计出与深度数据相吻合的三维姿态.与传统方法相比,本文方法不需要任何姿态数据库.实验表明,本文方法得到的结果更加精确.     

3D-freehand-pose initialization based on operator’s cognitive behavioral models

Tracking, recognition and interaction based on 3D freehand are a part of our virtual assembly system, in which monocular camera is used to input online freehand videos and the hand pose tracker requires a reliable initial pose in the first frame. A novel approach to initializing 3D pose and position of freehand is put forward in this paper visualization of 3D hand model and modeling the operators’ cognitive behaviors. Our approach is composed of three phases: hand posture recognition, coarse-tuning and fine-tuning. The operator moves his/her hand onto the to meet the needs of our virtual assembly system. The main contribution of this paper is that the three core techniques are for the first time integrated together, including human–computer interaction (HCI) in the process of initializing, projection of the 3D hand model in the period of coarse-tuning time. Then, the computer repeatedly fine-tunes the 3D hand model until the projection of the 3D hand model is completely superimposed onto the operator’s hand image. We focus on exploring and modeling cognitive behavior of operator’s hand upon which we design our initialization algorithm. Our research shows that cognitive behavioral models are not only beneficial to reducing cognitive loads for operators, because it makes the computers cater for the changes of the operators’ hand poses, but also helpful to address high dimensionality of articulated 3D hand model. Our experimental results also show that the approach presented in this paper is easier, more pleasurable and satisfactory experience for the operators. Our initialization system has successfully been applied to our 3D freehand tracking system and a simulation virtual assembly system.     

虚拟声音会议环境中的应用共享

应用共享系统是一种ＣＳＣＷ系统工具,它可以将单用户应用程序在多人的网络环境中协同地使用。在应用共享系统中需要提供各艇户间方便,直接的交互方式。     

视频会议中视频分割合成的研究

视频会议系统正在吸引越来越多在地域上分散的人们的注意。随着多媒体信息处理和通信技术的飞速发展，虚拟视频会议系统越来越成为可能。和传统的基于二维图像表述的视频会议系统比较，三维虚拟会议系统能为人们提供更好的以人为中心的交流。在这两种情况下，高效精确的视频分割和视图合成都是关键问题。就以下3方面的主要技术进行了研究：背景分离、视差估计以及视图合成。     

Automatic creation of a talking head from a video sequence

In this paper, a real-time system to create a talking head from a video sequence without any user intervention is presented. In the proposed system, a probabilistic approach, to decide whether or not extracted facial features are appropriate for creating a three-dimensional (3-D) face model, is presented. Automatically extracted two-dimensional facial features from a video sequence are fed into the proposed probabilistic framework before a corresponding 3-D face model is built to avoid generating an unnatural or nonrealistic 3-D face model. To extract face shape, we also present a face shape extractor based on an ellipse model controlled by three anchor points, which is accurate and computationally cheap. To create a 3-D face model, a least-square approach is presented to find a coefficient vector that is necessary to adapt a generic 3-D model into the extracted facial features. Experimental results show that the proposed system can efficiently build a 3-D face model from a video sequence without any user intervention for various Internet applications including virtual conference and a virtual story teller that do not require much head movements or high-quality facial animation.     

Adaptive user modelling in car racing games using behavioural and physiological data

Personalised content adaptation has great potential to increase user engagement in video games. Procedural generation of user-tailored content increases the self-motivation of players as they immerse themselves in the virtual world. An adaptive user model is needed to capture the skills of the player and enable automatic game content altering algorithms to fit the individual user. We propose an adaptive user modelling approach using a combination of unobtrusive physiological data to identify strengths and weaknesses in user performance in car racing games. Our system creates user-tailored tracks to improve driving habits and user experience, and to keep engagement at high levels. The user modelling approach adopts concepts from the Trace Theory framework; it uses machine learning to extract features from the user’s physiological data and game-related actions, and cluster them into low level primitives. These primitives are transformed and evaluated into higher level abstractions such as experience, exploration and attention. These abstractions are subsequently used to provide track alteration decisions for the player. Collection of data and feedback from 52 users allowed us to associate key model variables and outcomes to user responses, and to verify that the model provides statistically significant decisions personalised to the individual player. Tailored game content variations between users in our experiments, as well as the correlations with user satisfaction demonstrate that our algorithm is able to automatically incorporate user feedback in subsequent procedural content generation.     

A live video streaming system for intuitive human-system interaction

In this paper, we propose an intuitive live video streaming system based on virtual reality technologies among people who are far apart. This system is a kind of server-client system, and can provide remote users with virtual 3D audiovisual fields in real time via a very-high-speed network. The server captures audio and video data from its clients, compiles them into a 3D audiovisual scene at a virtual conference, and broadcasts it to the clients. At the present stage, our system captures 2 videos and creates one 3D video at a time. Our system can play 3D audiovisual contents on Windows XP systems as well as on CAVE systems. Currently, our system can play the 3D video contents at about 2.36 fps under a LAN environment. This work was presented in part at the 12th International Symposium on Artificial Life and Robotics, Oita, Japan, January 25–27, 2007     

基于SIP的视频会议中发言权管理的研究与实现

在互联网上,视频会议为了避免和解决用户之间的冲突以及相互牵连,需要协议支持的。视频会议中的冲突就是用户之间的资源共享冲突,一般采用的解决方法就是提供应用一些会议管理,如成员资格管理和发言权管理等。为了确保在视频会议应用中,能够使各用户无阻碍的共享“发言”这一资源,提出了一个基于SIP的视频会议中发言权管理的分布式算法思想。     

基于非线性流形学习的3维人体运动合成

为了实现3维人体运动的有效合成,提出了一种基于非线性流形学习的3维人体运动合成框架及算法,并可应用于方便、快捷、用户可控的3维人体运动合成。该合成算法框架先采用非线性流形降维方法将高维运动样本映射到低维流形上,同时求解其本征运动语义参数空间的表达,然后将用户在低维运动语义参数空间中交互生成的样本通过逆向映射重建得到具有新运动语义特征的3维运动序列。实验结果表明该方法不仅能够对运动物理参数（如特定关节的运动位置、物理运动特征）进行较为精确的控制,还可用于合成具有高层运动语义（运动风格）的新运动数据。与现有运动合成方法比较,该方法具有用户可控、交互性强等优点,能够应用于常见3维人体运动数据的高效生成。     

Synchronous learning best practices: An action research study

Low cost and significant advances in technology now allow instructors to create their own virtual learning environments. Creating social interactions within a virtual space that emulates the physical classroom remains challenging. While students are familiar with virtual worlds and video meetings, they are inexperienced as virtual learners. Over a nine year period we applied iterative cycles of action research through numerous large classes to systematically uncover attributes of success when executing synchronous learning in distributed environments. Findings show technology is not the source of problems; rather, difficulties emerge from human behaviors and their interactions with system features. We conclude with practical takeaway guidelines for video conferencing and immersive virtual environments and a model of nexus of control that elaborates software and classroom management attributes that can lead to successful execution.     

高层建筑内局域网系统设计与实现

本文通过采用核心层、汇聚层和接入层三层网络架构模式,实现高层建筑内局域网系统设计,整个设计由用户 需求分析、网络拓扑设计、IP 地址划分、虚拟局域网（VLAN）设计、网络冗余设计及综合布线设计六部分组成,满足用户通过网 络实现语音、数据、图形图像、视频等高效传送的要求。