Path planning directed motion control of virtual humans in complex environments

Natural motion synthesis of virtual humans have been studied extensively, however, motion control of virtual characters actively responding to complex dynamic environments is still a challenging task in computer animation. It is a labor and cost intensive animator-driven work to create realistic human motions of character animations in a dynamically varying environment in movies, television and video games. To solve this problem, in this paper we propose a novel approach of motion synthesis that applies the optimal path planning to direct motion synthesis for generating realistic character motions in response to complex dynamic environment. In our framework, SIPP (Safe Interval Path Planning) search is implemented to plan a globally optimal path in complex dynamic environments. Three types of control anchors to motion synthesis are for the first time defined and extracted on the obtained planning path, including turning anchors, height anchors and time anchors. Directed by these control anchors, highly interactive motions of virtual character are synthesized by motion field which produces a wide variety of natural motions and has high control agility to handle complex dynamic environments. Experimental results have proven that our framework is capable of synthesizing motions of virtual humans naturally adapted to the complex dynamic environments which guarantee both the optimal path and the realistic motion simultaneously.     

Statistics-based Motion Synthesis for Social Conversations

Plausible conversations among characters are required to generate the ambiance of social settings such as a restaurant, hotel lobby, or cocktail party. In this paper, we propose a motion synthesis technique that can rapidly generate animated motion for characters engaged in two-party conversations. Our system synthesizes gestures and other body motions for dyadic conversations that synchronize with novel input audio clips. Human conversations feature many different forms of coordination and synchronization. For example, speakers use hand gestures to emphasize important points, and listeners often nod in agreement or acknowledgment. To achieve the desired degree of realism, our method first constructs a motion graph that preserves the statistics of a database of recorded conversations performed by a pair of actors. This graph is then used to search for a motion sequence that respects three forms of audio-motion coordination in human conversations: coordination to phonemic clause, listener response, and partner's hesitation pause. We assess the quality of the generated animations through a user study that compares them to the originally recorded motion and evaluate the effects of each type of audio-motion coordination via ablation studies.     

A Comprehensive Review of Data-Driven Co-Speech Gesture Generation

Gestures that accompany speech are an essential part of natural and efficient embodied human communication. The automatic generation of such co-speech gestures is a long-standing problem in computer animation and is considered an enabling technology for creating believable characters in film, games, and virtual social spaces, as well as for interaction with social robots. The problem is made challenging by the idiosyncratic and non-periodic nature of human co-speech gesture motion, and by the great diversity of communicative functions that gestures encompass. The field of gesture generation has seen surging interest in the last few years, owing to the emergence of more and larger datasets of human gesture motion, combined with strides in deep-learning-based generative models that benefit from the growing availability of data. This review article summarizes co-speech gesture generation research, with a particular focus on deep generative models. First, we articulate the theory describing human gesticulation and how it complements speech. Next, we briefly discuss rule-based and classical statistical gesture synthesis, before delving into deep learning approaches. We employ the choice of input modalities as an organizing principle, examining systems that generate gestures from audio, text and non-linguistic input. Concurrent with the exposition of deep learning approaches, we chronicle the evolution of the related training data sets in terms of size, diversity, motion quality, and collection method (e.g., optical motion capture or pose estimation from video). Finally, we identify key research challenges in gesture generation, including data availability and quality; producing human-like motion; grounding the gesture in the co-occurring speech in interaction with other speakers, and in the environment; performing gesture evaluation; and integration of gesture synthesis into applications. We highlight recent approaches to tackling the various key challenges, as well as the limitations of these approaches, and point toward areas of future development.     

基于多分辨率模型的快速运动过渡生成算法

运动捕获技术可以记录人体关节运动的细节,是当前最有前景的计算机动画技术.然而,运动数据的重用性一直是个难点,为此,多种运动编辑手段被提出.运动过渡是一种常见的编辑技术,它可以将输入的两端运动序列拼接,形成新的运动序列.其中,过渡点选择的合理与否直接影响着结果运动的质量.在两运动间选择过渡点,需要对输入运动的每一对帧之间分别计算帧间的距离,其计算复杂度是O(n2)的,通过引入多分辨率模型,文中将该复杂度降低到O(nlog2n),同时试验结果表明,此方法并未损害到结果运动的质量.     

Walk the talk: coordinating gesture with locomotion for conversational characters

Communicative behaviors are a very important aspect of human behavior and deserve special attention when simulating groups and crowds of virtual pedestrians. Previous approaches have tended to focus on generating believable gestures for individual characters and talker‐listener behaviors for static groups. In this paper, we consider the problem of creating rich and varied conversational behaviors for data‐driven animation of walking and jogging characters. We captured ground truth data of participants conversing in pairs while walking and jogging. Our stylized splicing method takes as input a motion captured standing gesture performance and a set of looped full body locomotion clips. Guided by the ground truth metrics, we perform stylized splicing and synchronization of gesture with locomotion to produce natural conversations of characters in motion. Copyright © 2016 John Wiley & Sons, Ltd.     

Expressive facial animation synthesis by learning speech coarticulation and expression spaces

Synthesizing expressive facial animation is a very challenging topic within the graphics community. In this paper, we present an expressive facial animation synthesis system enabled by automated learning from facial motion capture data. Accurate 3D motions of the markers on the face of a human subject are captured while he/she recites a predesigned corpus, with specific spoken and visual expressions. We present a novel motion capture mining technique that "learns" speech coarticulation models for diphones and triphones from the recorded data. A phoneme-independent expression eigenspace (PIEES) that encloses the dynamic expression signals is constructed by motion signal processing (phoneme-based time-warping and subtraction) and principal component analysis (PCA) reduction. New expressive facial animations are synthesized as follows: First, the learned coarticulation models are concatenated to synthesize neutral visual speech according to novel speech input, then a texture-synthesis-based approach is used to generate a novel dynamic expression signal from the PIEES model, and finally the synthesized expression signal is blended with the synthesized neutral visual speech to create the final expressive facial animation. Our experiments demonstrate that the system can effectively synthesize realistic expressive facial animation     

一种基于传感器的人体上肢运动实时跟踪方法

实时跟踪人体运动是人机交互的重要研究课题,可以广泛应用于虚拟现实、虚拟人运动合成、聋人手语自动生成、计算机3D动画、机器人运动控制、远程人机交互等领域。文中介绍了一种基于传感器的人体上肢运动实时跟踪方法,给出了该方法的虚拟人模型、计算原理与校正方法,最后介绍了整个方法的实现以及在中国聋人手语自动合成中的应用。该方法具有使用传感器少,运动跟踪精度高、计算过程简单而且速度快等特点。     

模板化的人体运动合成

为解决现有运动合成方法中控制方式过于复杂的问题,提出一种模板化的运动合成模型,旨在降低运动合成技术的应用门槛.利用稀疏主成分分析(Sparse principal component analysis, SPCA)、Group lasso和Exclusive group lasso对人体运动进行建模,使其对应的每一个低维参数只依赖于少数几个人体关节,构成人体运动的一个内在自由度(Degree of freedom, DOF),并具有直观语义;同时,每个关节被尽量少的低维参数所控制,以减少低维参数对彼此所控制的自由度的交叉影响.实验表明,通过直观地修改低维参数,就能够实时地控制每个参数对应的摆臂幅度、踢腿高度、跳跃距离等运动属性.这种模板学习、模板定制的两步方法,有效地降低了运动合成控制的复杂度,即便非专业人员也可以用其进行艺术创作.     

Generating 3D architectural models based on hand motion and gesture

This paper presents a novel method for rapidly generating 3D architectural models based on hand motion and design gestures captured by a motion capture system. A set of sign language-based gestures, architectural hand signs (AHS), has been developed. AHS is performed on the left hand to define various “components of architecture”, while “location, size and shape” information is defined by the motion of Marker-Pen on the right hand. The hand gestures and motions are recognized by the system and then transferred into 3D curves and surfaces correspondingly. This paper demonstrates the hand gesture-aided architectural modeling method with some case studies.     

Constrained synthesis of textural motion for articulated characters

Obtaining high-quality, realistic motions of articulated characters is both time consuming and expensive, necessitating the development of easy-to-use and effective tools for motion editing and reuse. We propose a new simple technique for generating constrained variations of different lengths from an existing captured or otherwise animated motion. Our technique is applicable to textural motions, such as walking or dancing, where the motion sequence can be decomposed into shorter motion segments without an obvious temporal ordering among them. Inspired by previous work on texture synthesis and video textures, our method essentially produces a reordering of these shorter segments. Discontinuities are eliminated by carefully choosing the transition points and applying local adaptive smoothing in their vicinity, if necessary. The user is able to control the synthesis process by specifying a small number of simple constraints.     

Analysis of head gesture and prosody patterns for prosody-driven head-gesture animation

We propose a new two-stage framework for joint analysis of head gesture and speech prosody patterns of a speaker towards automatic realistic synthesis of head gestures from speech prosody. In the first stage analysis, we perform Hidden Markov Model (HMM) based unsupervised temporal segmentation of head gesture and speech prosody features separately to determine elementary head gesture and speech prosody patterns, respectively, for a particular speaker. In the second stage, joint analysis of correlations between these elementary head gesture and prosody patterns is performed using Multi-Stream HMMs to determine an audio-visual mapping model. The resulting audio-visual mapping model is then employed to synthesize natural head gestures from arbitrary input test speech given a head model for the speaker. In the synthesis stage, the audio-visual mapping model is used to predict a sequence of gesture patterns from the prosody pattern sequence computed for the input test speech. The Euler angles associated with each gesture pattern are then applied to animate the speaker head model. Objective and subjective evaluations indicate that the proposed synthesis by analysis scheme provides natural looking head gestures for the speaker with any input test speech, as well as in "prosody transplant" and gesture transplant" scenarios.     

Presence and interaction in mixed reality environments

In this paper, we present a simple and robust mixed reality (MR) framework that allows for real-time interaction with virtual humans in mixed reality environments under consistent illumination. We will look at three crucial parts of this system: interaction, animation and global illumination of virtual humans for an integrated and enhanced presence. The interaction system comprises of a dialogue module, which is interfaced with a speech recognition and synthesis system. Next to speech output, the dialogue system generates face and body motions, which are in turn managed by the virtual human animation layer. Our fast animation engine can handle various types of motions, such as normal key-frame animations, or motions that are generated on-the-fly by adapting previously recorded clips. Real-time idle motions are an example of the latter category. All these different motions are generated and blended on-line, resulting in a flexible and realistic animation. Our robust rendering method operates in accordance with the previous animation layer, based on an extended for virtual humans precomputed radiance transfer (PRT) illumination model, resulting in a realistic rendition of such interactive virtual characters in mixed reality environments. Finally, we present a scenario that illustrates the interplay and application of our methods, glued under a unique framework for presence and interaction in MR.     

基于时空约束的运动编辑和运动重定向

近年来兴起的运动捕获已成为人体动画中最有应用前景的技术之一，目前运动捕获手段很多，但是通常成本高，而且捕获到的运动类型比较单一，为了提高运动捕获数据的重用性，生成与复杂场景协调的多样的动画，必须对捕获的运动数据进行编辑和重定向处理，介绍了一种基于时空约束的运动编辑和运动重定向方法，通过规定一组时空约束条件，建立相应的目标函数，采用逆向运动学和数值优化方法求解出满足约束条件的运动姿势，实验结果表明，该方法可以生成多种满足不同场景婪泊逼真运动，提出了数据的重用性。     

One step at a time: animating virtual characters based on foot placement

Especially in a constrained virtual environment, precise control of foot placement during character locomotion is crucial to avoid collisions and to ensure a natural locomotion. In this paper, we present an extension of the step space: a novel technique for generating animations of a character walking over a set of desired foot steps in real time. We use an efficient greedy nearest-neighbor approach and warp the resulting animation so that it adheres to both spatial and temporal constraints. We will show that our technique can generate realistic locomotion animations over an input path very efficiently even though we impose many constraints on the animation. We also present a simple footstep planning technique that automatically plans regular stepping and sidestepping based on an input path with clearance information generated by a path planner.     

基于多自主智能体的群体动画创作

群体动画一直是计算机动画界一个具有挑战性的研究方向，提出了一个基于多自主智能体的群体动画创作框架：群体中的各角色作为自主智能体，能感知环境信息，产生意图，规划行为，最后通过运动系统产生运动来完成行为和实现意图，与传统的角色运动生成机理不同，首先采用运动捕获系统建立基本运动库，然后通过运动编辑技术对基本运动进行处理以最终得到角色运动，应用本技术，动画师只需“拍摄”角色群体的运动就能创作群体动画，极大地提高了制作效率。     

Fragment-based responsive character motion for interactive games

Fragment-based character animation has become popular in recent years. By stringing appropriate motion capture fragments together, the system drives characters responding to the control signals of the user and generates realistic character motions. In this paper, we propose a novel, straightforward and fast method to build the control policy table, which selects the next motion fragment to play based on the current user’s input and the previous motion fragment. During the synthesis of the control policy table, we cluster similar fragments together to create several fragment classes. Dynamic programming is employed to generate the training samples based on the control signals of the user. Finally, we use a supervised learning routine to create the tabular control policy. We demonstrate the efficacy of our method by comparing the motions generated by our controller to the optimal controller and other previous controllers. The results indicate that although a reinforcement learning algorithm known as value iteration also creates the tabular control policy, it is more complex and requires more expensive space–time cost in synthesis of the control policy table. Our approach is simple but efficient, and is practical for interactive character games.     

Reconstructing whole-body motions with wrist trajectories

Reconstructing whole-body motions using only a low-dimensional input reduces the cost of and efforts for performance capture significantly, and yet remains a challenging problem. We introduce a novel technique that synthesizes whole-body motion using the two wrist trajectories. Given the wrist trajectories, we first determine the optimal ankle trajectories from a large number of candidate ankle paths obtained from example poses in the motion database. The optimal trajectory is efficiently achieved by solving for the shortest path problem in a directed acyclic graph. Next, we use both the wrist and ankle trajectories as the low-dimensional control signals to achieve the whole-body pose at each time step. We show that our method can reconstruct various whole-body motions that can be recognized by arm motions, such as walking, stepping, and in-place upper-body motions. Comparisons with ground truth motions and with other methods are provided.     

基于小波变换的运动分析及其应用

对捕获的运动数据进行编辑处理 ,是生成新的复杂人体动画和提高运动捕获数据重用性的关键 ,但目前大多数运动编辑技术不具备对运动进行高层控制处理的能力 ,为此 ,提出了一种基于小波变换的运动编辑新算法 ,即将小波变换引入运动编辑 ,并对运动信号进行多分辨率分析 ,从而实现了运动特征增强、运动融合及运动特征提取与综合 .实验结果表明 ,该算法非常适合对运动特征进行处理 ,由于其能够在高层次上对运动进行有效的编辑 ,因而提高了动画师的工作效率 .     

基于深度神经网络的语音驱动发音器官的运动合成

实现一种基于深度神经网络的语音驱动发音器官运动合成的方法,并应用于语音驱动虚拟说话人动画合成. 通过深度神经网络(Deep neural networks, DNN)学习声学特征与发音器官位置信息之间的映射关系,系统根据输入的语音数据估计发音器官的运动轨迹,并将其体现在一个三维虚拟人上面. 首先,在一系列参数下对比人工神经网络(Artificial neural network, ANN)和DNN的实验结果,得到最优网络; 其次,设置不同上下文声学特征长度并调整隐层单元数,获取最佳长度; 最后,选取最优网络结构,由DNN 输出的发音器官运动轨迹信息控制发音器官运动合成,实现虚拟人动画. 实验证明,本文所实现的动画合成方法高效逼真.