Real-time estimation of hand gestures based on manifold learning from monocular videos

Object pose estimation by manifold learning has become a hot research area recently. In this paper, we propose an efficient method that can recover pose and viewpoints for numerous hand gestures from monocular videos based on Locality Preserving Projections. We first select some hand dynamic gestures as primitive hand motions and set a 3D-2D mapping table to relate 3D joint angles of sampling static pose with their projective silhouettes from arbitrary viewpoints. Then the embedding space and explicit mapping function are learnt for every primitive motion. In order to make classification and prediction among those embedding spaces, a Subspace Filtering Algorithm is also proposed which can recognize and recover numerous hand dynamic gestures by the combination of primitive gestures. At last, by using skin color cues and oriented k-Dops, multi-hands can be labeled and tracked separately and accurately. Extensive experimental results demonstrate qualitatively and quantitatively that 3D pose recovery of hands can be achieved by our method robustly and efficiently.     

Dynamic shape outlier detection for human locomotion

Dynamic human shape in video contains rich perceptual information, such as the body posture, identity, and even the emotional state of a person. Human locomotion activities, such as walking and running, have familiar spatiotemporal patterns that can easily be detected in arbitrary views. We present a framework for detecting shape outliers for human locomotion using a dynamic shape model that factorizes the body posture, the viewpoint, and the individual’s shape style. The model uses a common embedding of the kinematic manifold of the motion and factorizes the shape variability with respect to different viewpoints and shape styles in the space of the coefficients of the nonlinear mapping functions that are used to generate the shapes from the kinematic manifold representation. Given a corrupted input silhouette, an iterative procedure is used to recover the body posture, viewpoint, and shape style. We use the proposed outlier detection approach to fill in the holes in the input silhouettes, and detect carried objects, shadows, and abnormal motions.     

Recognition of dynamic hand gestures

This paper is concerned with the problem of recognition of dynamic hand gestures. We have considered gestures which are sequences of distinct hand poses. In these gestures hand poses can undergo motion and discrete changes. However, continuous deformations of the hand shapes are not permitted. We have developed a recognition engine which can reliably recognize these gestures despite individual variations. The engine also has the ability to detect start and end of gesture sequences in an automated fashion. The recognition strategy uses a combination of static shape recognition (performed using contour discriminant analysis), Kalman filter based hand tracking and a HMM based temporal characterization scheme. The system is fairly robust to background clutter and uses skin color for static shape recognition and tracking. A real time implementation on standard hardware is developed. Experimental results establish the effectiveness of the approach.     

利用图像距离的监督LLE头部姿态估计方法

局部线性嵌入(LLE)是一种经典流形学习方法,直接应用这种非监督的传统LLE估计图像中的头部姿态存在两点不足:未考虑图像像素空间信息和未利用样本标记信息.因此,本文结合图像欧式距离和偏置LLE流形学习方法,对头部姿态图像降维,并通过广义回归神经网络(GRNN)和多元线性回归的方法,估计头部图像的姿态.在FacePix头部姿态数据库的对比实验表明,本方法具有较好的头部姿态估计效果.     

基于双目视觉的人手定位与手势识别系统研究

提出了一种新的人手特征点提取方法,该方法将人手的质心作为匹配点,根据双目视觉定位数学模型计算目标位置信息,同时通过图像分割获取人手轮廓,利用轮廓凸包点特征来识别不同手势.在此基础上,研究设计了一种光学人手定位与手势识别系统,该系统在实时定位空间人手三维位置的同时,能够识别出相应的手势,可将其作为虚拟手的驱动接口,实现对虚拟物体的抓取、移动和释放操作.     

仿射不变的自适应局部线性嵌入

目的: 为将流形学习有效应用于图像的降维与识别中,并消除图像的仿射变换对流形结构产生的影响,本文提出一种仿射不变的自适应局部线性嵌入算法。方法: 该算法在局部线性嵌入的基础上,为适应产生各种仿射变换的图像样本,引入切线距离计算各样本之间的相似程度,以此描述样本空间中的距离,并通过图像相似度函数自适应计算样本空间中每一点的邻域数量。结果: 实验结果表明,该算法能够构造出更合理的低维流形结构,并有效提升统计识别的正确率。结论: 本文算法对仿射变换不敏感,表现出更强的稳健性。     

融合关节旋转特征和指尖距离特征的手势识别

作为人机交互的重要方式,手势交互和识别由于其具有的高自由度而成为计算机图形学、虚拟现实与人机交互等领域的研究热点.传统直接提取手势轮廓或手部关节点位置信息的手势识别方法,其提取的特征通常难以准确表示手势之间的区别.针对手势识别中不同手势具有的高自由度以及由于手势图像分辨率低、背景杂乱、手被遮挡、手指形状尺寸不同、个体差异性导致手势特征表示不准确等问题,本文提出了一种新的融合关节旋转特征和指尖距离特征的手势特征表示与手势识别方法.首先从手势深度图中利用手部模板并将手部看成链段结构提取手部20个关节点的3D位置信息;然后利用手部关节点位置信息提取四元数关节旋转特征和指尖距离特征,该表示构成了手势特征的内在表示;最后利用一对一支持向量机对手势进行有效识别分类.本文不仅提出了一种新的手势特征表示与提取方法,该表示融合了关节旋转信息和指尖距离特征;而且从理论上证明了该特征表示能唯一地表征手势关节点的位置信息;同时提出了基于一对一SVM多分类策略进行手势分类与识别.对ASTAR静态手势深度图数据集中8类中国数字手势和21类美国字母手势数据集分别进行了实验验证,其分类识别准确率分别为99.71%和85.24%.实验结果表明,本文提出的基于关节旋转特征和指尖距离特征的融合特征能很好地表示不同手势的几何特征,能准确地表征静态手势并进行手势识别.     

基于相对流形的局部线性嵌入

局部线性嵌入算法极大地依赖于邻域是否真实地反映了流形的内在结构,现有方法构造的邻域结构是拓扑不稳定的,对噪音和稀疏数据敏感.根据认知的相对性规律提出了相对变换,并用其构造了相对空间和相对流形.相对变换可以提高数据之间的可区分性,并能抑制噪音和数据稀疏的影响.在构造的相对空间和相对流形上确定数据点的邻域能够更真实地反映流形的内在结构,由此提出了增强的局部线性嵌入算法,明显地提高了性能,特别是基于流形的方法还同时提高了速度.标准数据集上的实验结果验证了该方法的有效性.     

流形距离与压缩感知核稀疏投影的局部线性嵌入算法

流形学习算法的目的是发现嵌入在高维数据空间中的低维表示,现有的流形学习算法对邻域参数k和噪声比较敏感。针对此问题,文中提出一种流形距离与压缩感知核稀疏投影的局部线性嵌入算法,其核心思想是集成局部线性嵌入算法对高维流形结构数据的降维有效性与压缩感知核稀疏投影的强鉴别性,以实现高效有降噪流形学习。首先,在选择各样本点的近邻域时,采用流形距离代替欧氏距离度量数据间相似度的方法,创建能够正确反映流形内部结构的邻域图,解决以欧氏距离作为相似性度量时对邻域参数的敏感。其次,利用压缩感知核稀疏投影作为从高维观测空间到低维嵌入空间的映射,增强算法的鉴别性。最后,利用Matlab工具对实验数据集进行仿真,进一步验证所提算法的有效性。     

Articulated tracking with manifold regularized particle filter

In this paper, we investigate articulated human motion tracking from video sequences using Bayesian approach. We derive a generic particle-based filtering procedure with a low-dimensional manifold. The manifold can be treated as a regularizer that enforces a distribution over poses during tracking process to be concentrated around the low-dimensional embedding. We refer to our method as manifold regularized particle filter. We present a particular implementation of our method based on back-constrained gaussian process latent variable model and gaussian diffusion. The proposed approach is evaluated using the real-life benchmark dataset HumanEva. We show empirically that the presented sampling scheme outperforms sampling-importance resampling and annealed particle filter procedures.     

多点触控的沙画手势识别

针对多点触控手势间接指令问题,提出了基于多点触控的沙画手势识别系统,该识别系统由时间、空间、形状信息控制。提出一种手势图形建模方法,测量手势的笔划之间的空间和时间关系。采用聚类算法标记手势图形中笔划的形状信息作为局部形状特征;利用基准方法HBF49特征提取全局形状特征。通过一组有10种不同多点触控的沙画手势的数据集评估基于多点触控的沙画手势识别系统,使用图嵌入方法和SVM分类进行手势识别,识别的准确率达到94.75%。实验结果证明,此研究对完成基于多点触控的沙画虚拟系统有重要作用。     

Adaptive visual gesture recognition for human–robot interaction using a knowledge-based software platform

In human–human communication we can adapt or learn new gestures or new users using intelligence and contextual information. Achieving natural gesture-based interaction between humans and robots, the system should be adaptable to new users, gestures and robot behaviors. This paper presents an adaptive visual gesture recognition method for human–robot interaction using a knowledge-based software platform. The system is capable of recognizing users, static gestures comprised of the face and hand poses, and dynamic gestures of face in motion. The system learns new users, poses using multi-cluster approach, and combines computer vision and knowledge-based approaches in order to adapt to new users, gestures and robot behaviors. In the proposed method, a frame-based knowledge model is defined for the person-centric gesture interpretation and human–robot interaction. It is implemented using the frame-based Software Platform for Agent and Knowledge Management (SPAK). The effectiveness of this method has been demonstrated by an experimental human–robot interaction system using a humanoid robot ‘Robovie’.     

Unsupervised image matching based on manifold alignment

This paper challenges the issue of automatic matching between two image sets with similar intrinsic structures and different appearances, especially when there is no prior correspondence. An unsupervised manifold alignment framework is proposed to establish correspondence between data sets by a mapping function in the mutual embedding space. We introduce a local similarity metric based on parameterized distance curves to represent the connection of one point with the rest of the manifold. A small set of valid feature pairs can be found without manual interactions by matching the distance curve of one manifold with the curve cluster of the other manifold. To avoid potential confusions in image matching, we propose an extended affine transformation to solve the nonrigid alignment in the embedding space. The comparatively tight alignments and the structure preservation can be obtained simultaneously. The point pairs with the minimum distance after alignment are viewed as the matchings. We apply manifold alignment to image set matching problems. The correspondence between image sets of different poses, illuminations, and identities can be established effectively by our approach.     

复杂环境中多信息融合的手势识别

针对复杂环境中的手势识别问题,提出了一种融合深度信息和红外信息的手势识别方法。首先利用Kinect摄像头的深度信息进行动态实时手势分割,然后融合红外图像复原手势区域。解决了实时手势分割和利用手势的空间分布特征进行手势识别时由于分割的手势区域有缺损或有人脸干扰时识别率低的问题。经实验验证,提出的方法不仅不受环境光线的影响,而且可以识别区分度较小的手势,对旋转、缩放、平移的手势识别也具有鲁棒性。对于区分度较大的手势,识别率高达100%。     

正交化近邻关系保持的降维及分类算法

针对近邻关系保持嵌入（NPE）算法易于受到降低后的维数影响,而且性能依赖于正确的维数估计的问题,提出了一种正交化的近邻关系保持的嵌入降维方法——ONPE。ONPE方法是使用数据点间的近邻关系来构造邻接图,假设每个数据点都能由其近邻点的线性组合表示,则可以通过提取数据点的局部几何信息,并在降维中保持提取的局部几何信息,迭代地计算正交基来得到数据的低维嵌入坐标。同时,在ONPE算法的基础上,利用局部几何信息,提出了一种在低维空间中使用标签传递（LNP）的分类算法——ONPC。其是假设高维空间中的局部近邻关系在降维后的空间中依然得到保持,并且数据点的类别可由近邻点的类别得到。在人工数据和人脸数据上的实验表明,该算法在减少维数依赖的同时,能有效提高NPE算法的分类性能。     

一种基于边界约束的流形展开方法

在流形学习的谱方法中, 流形展开被表述为优化问题. 这些优化问题的解是退化的, 即所有的样本将被嵌入到同一个点. 为了避免退化解, 谱方法对嵌入坐标人为地强加了一个单位协方差矩阵约束. 然而, 该约束往往导致流形展开的失真非常明显. 本文提出一种新的流形学习方法, 彻底抛弃了人为的单位协方差矩阵约束. 主要思路是先对流形边界进行嵌入, 然后再求流形内部的嵌入; 流形边界的嵌入位置被确定后, 流形内部样本的嵌入位置将被边界拉开, 使得它们不会都收缩到一个点上, 从而避免了退化解的出现. 将流形边界的嵌入位置作为边界条件, 求解一个线性方程组来得到内部样本的嵌入; 该线性方程组反映了尽量保持邻近样本间距离不变的要求. 流形边界的嵌入由简化流形的嵌入求出; 为此, 本文还设计了一种流形边界检测算法以及一种流形简化算法. 与目前代表性的几种流形学习方法进行了比较实验, 结果表明了本文方法的有效性, 其展开失真比谱方法明显要小.     

Static hand gesture recognition using neural networks

This paper presents a novel technique for hand gesture recognition through human–computer interaction based on shape analysis. The main objective of this effort is to explore the utility of a neural network-based approach to the recognition of the hand gestures. A unique multi-layer perception of neural network is built for classification by using back-propagation learning algorithm. The goal of static hand gesture recognition is to classify the given hand gesture data represented by some features into some predefined finite number of gesture classes. The proposed system presents a recognition algorithm to recognize a set of six specific static hand gestures, namely: Open, Close, Cut, Paste, Maximize, and Minimize. The hand gesture image is passed through three stages, preprocessing, feature extraction, and classification. In preprocessing stage some operations are applied to extract the hand gesture from its background and prepare the hand gesture image for the feature extraction stage. In the first method, the hand contour is used as a feature which treats scaling and translation of problems (in some cases). The complex moment algorithm is, however, used to describe the hand gesture and treat the rotation problem in addition to the scaling and translation. The algorithm used in a multi-layer neural network classifier which uses back-propagation learning algorithm. The results show that the first method has a performance of 70.83% recognition, while the second method, proposed in this article, has a better performance of 86.38% recognition rate.     

Robust locally linear embedding

In the past few years, some nonlinear dimensionality reduction (NLDR) or nonlinear manifold learning methods have aroused a great deal of interest in the machine learning community. These methods are promising in that they can automatically discover the low-dimensional nonlinear manifold in a high-dimensional data space and then embed the data points into a low-dimensional embedding space, using tractable linear algebraic techniques that are easy to implement and are not prone to local minima. Despite their appealing properties, these NLDR methods are not robust against outliers in the data, yet so far very little has been done to address the robustness problem. In this paper, we address this problem in the context of an NLDR method called locally linear embedding (LLE). Based on robust estimation techniques, we propose an approach to make LLE more robust. We refer to this approach as robust locally linear embedding (RLLE). We also present several specific methods for realizing this general RLLE approach. Experimental results on both synthetic and real-world data show that RLLE is very robust against outliers.     

一种基于抓取簇和碰撞体素的工业零件抓取姿态检测算法

针对工业上常见的散乱堆叠零件的抓取问题,提出一种基于抓取簇和碰撞体素的抓取姿态检测算法。所提出的抓取簇是定义在零件上的连续抓取姿态集合,解决了传统方法中因采用离散固定抓取点而导致可抓取点丢失、筛选效率低的问题。先对料箱和场景点云进行体素化;然后把包含料箱或点云的体素标记为碰撞体素,并把与碰撞体素相邻的体素标记为风险体素,从而建立体素化的碰撞模型;接下来,根据抓取簇的几何性质计算出候选的抓取姿态及其对应的抓取路径;最后,通过检测抓取路径所经过的体素类型来完成快速的碰撞检测,从而筛选出最优抓取姿态。基于所提算法搭建了完整的Bin-Picking系统,并对多种实际工业场景中常见的零件进行仿真实验和实际抓取实验,结果表明：该算法能够快速、准确地检测出安全的抓取姿态,实际抓取的平均成功率达92.2%,料箱平均清空率达87.2%,较传统方法有明显提升,且抓取过程均未发生碰撞,可满足实际工业应用的要求。     

基于最优文档嵌入的《红楼梦》作者辨析

该文以包括《红楼梦》在内的51部当代及明清文学作品为语料集,利用文档嵌入算法,根据文档嵌入向量的酉不变性定义了不同作者作品文档嵌入矩阵及文档嵌入损失函数,构建了文档嵌入模型中最优维度及最优窗口的选择模型,并根据文本用词和文档主题语义特征构建了高维空间中的文档嵌入向量。通过无监督的流形学习降维映射以及有监督的分类算法多组实验,验证了通过文档嵌入得到的向量空间模型可以有效区分不同作者的写作风格,对于已知确定作者的作品分类准确率达99.6%,对于风格较为接近的作者也可以有效识别,例如,文风相似的路遥和陈忠实。并在此分类模型的基础上,构建了变尺度滑动窗口分类模型对《红楼梦》进行深入分析,印证了“红楼梦”前80回与后40回可能来自不同作者,还发现了前100回与后20回也存在着较大的风格差异,不排除有再次更换作者的可能。该文在计算机技术层面上为《红楼梦》的作者辨析问题提供了一种支持意见和新的见解。