Ongoing human action recognition with motion capture

Ongoing human action recognition is a challenging problem that has many applications, such as video surveillance, patient monitoring, human–computer interaction, etc. This paper presents a novel framework for recognizing streamed actions using Motion Capture (MoCap) data. Unlike the after-the-fact classification of completed activities, this work aims at achieving early recognition of ongoing activities. The proposed method is time efficient as it is based on histograms of action poses, extracted from MoCap data, that are computed according to Hausdorff distance. The histograms are then compared with the Bhattacharyya distance and warped by a dynamic time warping process to achieve their optimal alignment. This process, implemented by our dynamic programming-based solution, has the advantage of allowing some stretching flexibility to accommodate for possible action length changes. We have shown the success and effectiveness of our solution by testing it on large datasets and comparing it with several state-of-the-art methods. In particular, we were able to achieve excellent recognition rates that have outperformed many well known methods.     

Averaging of motion capture recordings for movements’ templates generation

In this paper we propose, describe and evaluate the novel motion capture (MoCap) data averaging framework. It incorporates hierarchical kinematic model, angle coordinates’ preprocessing methods, that recalculate the original MoCap recording making it applicable for further averaging algorithms, and finally signals averaging processing. We have tested two signal averaging methods namely Kalman Filter (KF) and Dynamic Time Warping barycenter averaging (DBA). The propose methods have been tested on MoCap recordings of elite Karate athlete, multiple champion of Oyama karate knockdown kumite who performed 28 different karate techniques repeated 10 times each. The proposed methods proved to have not only high effectiveness measured with root-mean-square deviation (4.04 ± 5.03 degrees for KF and 5.57 ± 6.27 for DBA) and normalized Dynamic Time Warping distance (0.90 ± 1.58 degrees for KF and 0.93 ± 1.23 for DBA), but also the reconstruction and visualization of those recordings persists all crucial aspects of those complicated actions. The proposed methodology has many important applications in classification, clustering, kinematic analysis and coaching. Our approach generates an averaged full body motion template that can be practically used for example for human actions recognition. In order to prove it we have evaluated templates generated by our method in human action classification tasks using DTW classifier. We have made two experiments. In first leave - one - out cross - validation we have obtained 100% correct recognitions. In second experiment when we classified recordings of one person using templates of another recognition rate 94.2% was obtained.     

Multiresolution coding of motion capture data for real-time multimedia applications

Multimedia Tools and Applications - In this work, we present a novel and efficient method for coding of motion capture (MoCap) data obtained from recording of human actions. MoCap data is...     

时空域融合的骨架动作识别与交互研究

在人体骨架结构动作识别方法中,很多研究工作在提取骨架结构上的空间信息和运动信息后进行融合,没有对具有复杂时空关系的人体动作进行高效表达。本文提出了基于姿态运动时空域融合的图卷积网络模型(PM-STFGCN)。对于在时域上存在大量的干扰信息,定义了一种基于局部姿态运动的时域关注度模块(LPM-TAM),用于抑制时域上的干扰并学习运动姿态的表征。设计了基于姿态运动的时空域融合模块(PM-STF),融合时域运动和空域姿态特征并进行自适应特征增强。通过实验验证,本文提出的方法是有效性的,与其他方法相比,在识别效果上具有很好的竞争力。设计的人体动作交互系统,验证了在实时性和准确率上优于语音交互系统。     

View invariant action recognition using weighted fundamental ratios

In this paper, we fully investigate the concept of fundamental ratios, demonstrate their application and significance in view-invariant action recognition, and explore the importance of different body parts in action recognition. A moving plane observed by a fixed camera induces a fundamental matrix F between two frames, where the ratios among the elements in the upper left 2 × 2 submatrix are herein referred to as the fundamental ratios. We show that fundamental ratios are invariant to camera internal parameters and orientation, and hence can be used to identify similar motions of line segments from varying viewpoints. By representing the human body as a set of points, we decompose a body posture into a set of line segments. The similarity between two actions is therefore measured by the motion of line segments and hence by their associated fundamental ratios. We further investigate to what extent a body part plays a role in recognition of different actions and propose a generic method of assigning weights to different body points. Experiments are performed on three categories of data: the controlled CMU MoCap dataset, the partially controlled IXMAS data, and the more challenging uncontrolled UCF-CIL dataset collected on the internet. Extensive experiments are reported on testing (i) view-invariance, (ii) robustness to noisy localization of body points, (iii) effect of assigning different weights to different body points, (iv) effect of partial occlusion on recognition accuracy, and (v) determining how soon our method recognizes an action correctly from the starting point of the query video.     

Model-based recognition of human actions by trajectory matching in phase spaces

This paper presents a human action recognition framework based on the theory of nonlinear dynamical systems. The ultimate aim of our method is to recognize actions from multi-view video. We estimate and represent human motion by means of a virtual skeleton model providing the basis for a view-invariant representation of human actions. Actions are modeled as a set of weighted dynamical systems associated to different model variables. We use time-delay embeddings on the time series resulting of the evolution of model variables along time to reconstruct phase portraits of appropriate dimensions. These phase portraits characterize the underlying dynamical systems. We propose a distance to compare trajectories within the reconstructed phase portraits. These distances are used to train SVM models for action recognition. Additionally, we propose an efficient method to learn a set of weights reflecting the discriminative power of a given model variable in a given action class. Our approach presents a good behavior on noisy data, even in cases where action sequences last just for a few frames. Experiments with marker-based and markerless motion capture data show the effectiveness of the proposed method. To the best of our knowledge, this contribution is the first to apply time-delay embeddings on data obtained from multi-view video.     

A group of novel approaches and a toolkit for motion capture data reusing

Now more and more motion capture (MoCap) systems are used to acquire realistic and highly detailed motion data which are widely used for producing animations of human-like characters in a variety of applications, such as simulations, video games and animation films. And recently large MoCap databases are available. As a kind of emerging multimedia data, 3D human motion has its own specific data form and standard format. But to the best of our knowledge, only a few approaches have been explored for 3D MoCap data feature representation and reusing. This paper proposes a group of novel approaches for posture feature representation, motion sequence segmentation, key-frame extraction and content-based motion retrieval, which are all very important for MoCap data reusing and benefit to the efficient animation production. To validate these approaches, we set up a MoCap database and implemented a prototype toolkit. The experiments show that the proposed algorithms could achieve the approvable results.     

A novel approach for recognition of human actions with semi-global features

In this study a new approach is presented for the recognition of human actions of everyday life with a fixed camera. The originality of the presented method consists in characterizing sequences by a temporal succession of semi-global features, which are extracted from “space-time micro-volumes”. The advantage of this approach lies in the use of robust features (estimated on several frames) associated with the ability to manage actions with variable durations and easily segment the sequences with algorithms that are specific to time-varying data. Each action is actually characterized by a temporal sequence that constitutes the input of a Hidden Markov Model system for the recognition. Results presented of 1,614 sequences performed by several persons validate the proposed approach.     

基于动作图的视角无关动作识别

针对视角无关的动作识别,提出加权字典向量描述方法和动作图识别模型.将视频中的局部兴趣点特征和全局形状描述有机结合,形成加权字典向量的描述方法,该方法既具有兴趣点抗噪声强的优点,又可克服兴趣点无法识别静态动作的缺点.根据运动捕获、点云等三维运动数据构建能量曲线,提取关键姿势,生成基本运动单元,并通过自连接、向前连接和向后连接3种连接方式构成有向图,称为本质图.本质图向各个方向投影,根据节点近邻规则建立的有向图称为动作图.通过Na?ve Bayes训练动作图模型,采用Viterbi算法计算视频与动作图的匹配度,根据最大匹配度标定视频序列.动作图具有多角度投影和投影平滑过渡等特点,因此可识别任意角度、任意运动方向的视频序列.实验结果表明,该算法具有较好的识别效果,可识别单目视频、多目视频和多动作视频.     

Exploring trace transform for robust human action recognition

Machine based human action recognition has become very popular in the last decade. Automatic unattended surveillance systems, interactive video games, machine learning and robotics are only few of the areas that involve human action recognition. This paper examines the capability of a known transform, the so-called Trace, for human action recognition and proposes two new feature extraction methods based on the specific transform. The first method extracts Trace transforms from binarized silhouettes, representing different stages of a single action period. A final history template composed from the above transforms, represents the whole sequence containing much of the valuable spatio-temporal information contained in a human action. The second, involves Trace for the construction of a set of invariant features that represent the action sequence and can cope with variations usually appeared in video capturing. The specific method takes advantage of the natural specifications of the Trace transform, to produce noise robust features that are invariant to translation, rotation, scaling and are effective, simple and fast to create. Classification experiments performed on two well known and challenging action datasets (KTH and Weizmann) using Radial Basis Function (RBF) Kernel SVM provided very competitive results indicating the potentials of the proposed techniques.     

基于信道状态信息的非接触式人员动作识别方法

传感器与摄像头等设备的传统动作识别存在受环境影响大及侵犯用户隐私等问题,以京剧动作为研究对象,提出一种非接触式人员动作识别方法Wi-Opera。在离线阶段采集Wi-Fi路由设备上人体动作的信道状态信息（CSI）数据,利用巴特沃斯低通滤波器和小波变换方法对CSI数据分别进行去噪和平滑处理,通过主成分分析算法提取动作的特征值构建每个京剧动作的决策树,最终形成随机森林模型。在在线阶段实时采集的动作数据经过处理后,将京剧动作的特征值输入随机森林模型中进行识别,从而输出识别结果。实验结果表明,Wi-Opera方法的综合识别精度为94.6%,具有较高的识别精度和较强的鲁棒性。     

基于轻量级图卷积的人体骨架动作识别方法

视频中的人体动作识别在计算机视觉领域得到广泛关注,基于人体骨架的动作识别方法可以明确地表现人体动作,因此已逐渐成为该领域的重要研究方向之一。针对多数主流人体动作识别方法网络参数量大、计算复杂度高等问题,设计一种融合多流数据的轻量级图卷积网络,并将其应用于人体骨架动作识别任务。在数据预处理阶段,利用多流数据融合方法对4种特征数据流进行融合,通过一次训练就可得到最优结果,从而降低网络参数量。设计基于图卷积网络的非局部网络模块,以捕获图像的全局信息从而提高动作识别准确率。在此基础上,设计空间Ghost图卷积模块和时间Ghost图卷积模块,从网络结构上进一步降低网络参数量。在动作识别数据集NTU60 RGB+D和NTU120 RGB+D上进行实验,结果表明,与近年主流动作识别方法ST-GCN、2s AS-GCN、2s AGCN等相比,基于该轻量级图卷积网络的人体骨架动作识别方法在保持较低网络参数量的情况下能够取得较高的识别准确率。     

Discriminative human action recognition in the learned hierarchical manifold space

In this paper, we propose a hierarchical discriminative approach for human action recognition. It consists of feature extraction with mutual motion pattern analysis and discriminative action modeling in the hierarchical manifold space. Hierarchical Gaussian Process Latent Variable Model (HGPLVM) is employed to learn the hierarchical manifold space in which motion patterns are extracted. A cascade CRF is also presented to estimate the motion patterns in the corresponding manifold subspace, and the trained SVM classifier predicts the action label for the current observation. Using motion capture data, we test our method and evaluate how body parts make effect on human action recognition. The results on our test set of synthetic images are also presented to demonstrate the robustness.     

Forward non-rigid motion tracking for facial MoCap

For the existing motion capture (MoCap) data processing methods, manual interventions are always inevitable, most of which are derived from the data tracking process. This paper addresses the problem of tracking non-rigid 3D facial motions from sequences of raw MoCap data in the presence of noise, outliers and long time missing. We present a novel dynamic spatiotemporal framework to automatically solve the problem. First, based on a 3D facial topological structure, a sophisticated non-rigid motion interpreter (SNRMI) is put forward; together with a dynamic searching scheme, it cannot only track the non-missing data to the maximum extent but recover missing data (it can accurately recover more than five adjacent markers under long time (about 5 seconds) missing) accurately. To rule out wrong tracks of the markers labeled in open structures (such as mouth, eyes), a semantic-based heuristic checking method was raised. Second, since the existing methods have not taken the noise propagation problem into account, a forward processing framework is presented to solve the problem. Another contribution is the proposed method could track facial non-rigid motions automatically and forward, and is believed to greatly reduce even eliminate the requirements of human interventions during the facial MoCap data processing. Experimental results proved the effectiveness, robustness and accuracy of our system.     

Real-time Unsupervised Segmentation of human whole-body motion and its application to humanoid robot acquisition of motion symbols

An interactive loop between motion recognition and motion generation is a fundamental mechanism for humans and humanoid robots. We have been developing an intelligent framework for motion recognition and generation based on symbolizing motion primitives. The motion primitives are encoded into Hidden Markov Models (HMMs), which we call “motion symbols”. However, to determine the motion primitives to use as training data for the HMMs, this framework requires a manual segmentation of human motions. Essentially, a humanoid robot is expected to participate in daily life and must learn many motion symbols to adapt to various situations. For this use, manual segmentation is cumbersome and impractical for humanoid robots. In this study, we propose a novel approach to segmentation, the Real-time Unsupervised Segmentation (RUS) method, which comprises three phases. In the first phase, short human movements are encoded into feature HMMs. Seamless human motion can be converted to a sequence of these feature HMMs. In the second phase, the causality between the feature HMMs is extracted. The causality data make it possible to predict movement from observation. In the third phase, movements having a large prediction uncertainty are designated as the boundaries of motion primitives. In this way, human whole-body motion can be segmented into a sequence of motion primitives. This paper also describes an application of RUS to AUtonomous Symbolization of motion primitives (AUS). Each derived motion primitive is classified into an HMM for a motion symbol, and parameters of the HMMs are optimized by using the motion primitives as training data in competitive learning. The HMMs are gradually optimized in such a way that the HMMs can abstract similar motion primitives. We tested the RUS and AUS frameworks on captured human whole-body motions and demonstrated the validity of the proposed framework.