融合关节旋转特征和指尖距离特征的手势识别

作为人机交互的重要方式,手势交互和识别由于其具有的高自由度而成为计算机图形学、虚拟现实与人机交互等领域的研究热点.传统直接提取手势轮廓或手部关节点位置信息的手势识别方法,其提取的特征通常难以准确表示手势之间的区别.针对手势识别中不同手势具有的高自由度以及由于手势图像分辨率低、背景杂乱、手被遮挡、手指形状尺寸不同、个体差异性导致手势特征表示不准确等问题,本文提出了一种新的融合关节旋转特征和指尖距离特征的手势特征表示与手势识别方法.首先从手势深度图中利用手部模板并将手部看成链段结构提取手部20个关节点的3D位置信息;然后利用手部关节点位置信息提取四元数关节旋转特征和指尖距离特征,该表示构成了手势特征的内在表示;最后利用一对一支持向量机对手势进行有效识别分类.本文不仅提出了一种新的手势特征表示与提取方法,该表示融合了关节旋转信息和指尖距离特征;而且从理论上证明了该特征表示能唯一地表征手势关节点的位置信息;同时提出了基于一对一SVM多分类策略进行手势分类与识别.对ASTAR静态手势深度图数据集中8类中国数字手势和21类美国字母手势数据集分别进行了实验验证,其分类识别准确率分别为99.71%和85.24%.实验结果表明,本文提出的基于关节旋转特征和指尖距离特征的融合特征能很好地表示不同手势的几何特征,能准确地表征静态手势并进行手势识别.     

Nonlinear multiscale Maximal Lyapunov Exponent for accurate myoelectric signal classification

Surface Electromyography (sEMG) is a non-invasive, easy to record signal of superficial muscles from the skin surface. The sEMG is widely used in evaluating the functional status of the hand to assist in hand gesture recognition, prosthetics and rehabilitation applications. Considering the nonlinear and non-stationary characteristics of sEMG, hand gesture recognition using sEMG signals necessitate designers to use Maximal Lyapunov Exponent (MLE) or ensemble Empirical Mode Decomposition (EMD) based MLEs. In this research, we propose a hand gesture recognition method of sEMG based on nonlinear multiscale MLE. The aim is to increase the classification accuracy of sEMG features while reducing the complexity of EMD. The nonlinear MLE features are classified using Flexible Neural Tree (FNT), which can solve highly structured dependent problems of the Artificial Neural Network (ANN). The testing has been conducted using several experiments with five participants. The classification performance of nonlinear multiscale MLE method is compared with MLE and EMD-based MLE through simulations. Experimental results demonstrate that the former algorithm outperforms the two latter algorithms and can classify six different hand gestures up to 97.6% accuracy.     

基于多特征加权融合的静态手势识别

针对人工提取特征的单一性及卷积神经网络提取特征的遗漏性问题,提出了一种基于多特征加权融合的静态手势识别方法.首先,提取分割后的手势图像的傅里叶和Hu矩等形状特征,将两者融合作为手势图像的局部特征;设计双通道卷积神经网络提取手势图像的深层次特征,采用主成分分析方法对提取的特征进行降维;然后,将提取的局部特征和深层次特征进行加权融合作为手势识别的有效特征描述;最后,使用Softmax分类器进行手势图像分类.实验结果验证了提出方法的有效性,在手势图像数据集上的识别准确率达到了99%以上.     

Visual Recognition of Static/Dynamic Gesture: Gesture-Driven Editing System

This paper presents the visual recognition of static gesture (SG) or dynamic gesture (DG). Gesture is one of the most natural interface tools for human–computer interaction (HCI) as well as for communication between human beings. In order to implement a human-like interface, gestures could be recognized using only visual information such as the visual mechanism of human beings; SGs and DGs can be processed concurrently as well. This paper aims at recognizing hand gestures obtained from the visual images on a 2D image plane, without any external devices. Gestures are spotted by a task-specific state transition based on natural human articulation. SGs are recognized using image moments of hand posture, while DGs are recognized by analyzing their moving trajectories on the hidden Markov models (HMMs). We have applied our gesture recognition approach to gesture-driven editing systems operating in real time.     

基于手势识别的人机交互发展研究

近年来手势识别技术的快速发展,基于手势识别技术的人机交互应用系统的建立使得人机交互的发展前景广阔.从手形、手势和手形手势的建模出发,介绍了模板匹配、特征提取、神经网络和隐马尔可夫模型4种手势识别的方法,并且综述了基于手势识别技术人机交互的发展,详细介绍了3类人机交互系统：漫游型系统、编辑型系统和操作型系统.     

一种面向实时交互的变形手势跟踪方法

变形手势跟踪是基于视觉的人机交互研究中的一项重要内容.单摄像头条件下,提出一种新颖的变形手势实时跟踪方法.利用一组2D手势模型替代高维度的3D手模型.首先利用贝叶斯分类器对静态手势进行识别,然后对图像进行手指和指尖定位,通过将图像特征与识别结果进行匹配,实现了跟踪过程的自动初始化.提出将K-means聚类算法与粒子滤波相结合,用于解决多手指跟踪问题中手指互相干扰的问题.跟踪过程中进行跟踪状态检测,实现了自动恢复跟踪及手势模型更新.实验结果表明,该方法可以实现对变形手势快速、准确的连续跟踪,能够满足基于视觉的实时人机交互的要求.     

Static hand gesture recognition using neural networks

This paper presents a novel technique for hand gesture recognition through human–computer interaction based on shape analysis. The main objective of this effort is to explore the utility of a neural network-based approach to the recognition of the hand gestures. A unique multi-layer perception of neural network is built for classification by using back-propagation learning algorithm. The goal of static hand gesture recognition is to classify the given hand gesture data represented by some features into some predefined finite number of gesture classes. The proposed system presents a recognition algorithm to recognize a set of six specific static hand gestures, namely: Open, Close, Cut, Paste, Maximize, and Minimize. The hand gesture image is passed through three stages, preprocessing, feature extraction, and classification. In preprocessing stage some operations are applied to extract the hand gesture from its background and prepare the hand gesture image for the feature extraction stage. In the first method, the hand contour is used as a feature which treats scaling and translation of problems (in some cases). The complex moment algorithm is, however, used to describe the hand gesture and treat the rotation problem in addition to the scaling and translation. The algorithm used in a multi-layer neural network classifier which uses back-propagation learning algorithm. The results show that the first method has a performance of 70.83% recognition, while the second method, proposed in this article, has a better performance of 86.38% recognition rate.     

A feature covariance matrix with serial particle filter for isolated sign language recognition

As is widely recognized, sign language recognition is a very challenging visual recognition problem. In this paper, we propose a feature covariance matrix based serial particle filter for isolated sign language recognition. At the preprocessing stage, the fusion of the median and mode filters is employed to extract the foreground and thereby enhances hand detection. We propose to serially track the hands of the signer, as opposed to tracking both hands at the same time, to reduce the misdirection of target objects. Subsequently, the region around the tracked hands is extracted to generate the feature covariance matrix as a compact representation of the tracked hand gesture, and thereby reduce the dimensionality of the features. In addition, the proposed feature covariance matrix is able to adapt to new signs due to its ability to integrate multiple correlated features in a natural way, without any retraining process. The experimental results show that the hand trajectories as obtained through the proposed serial hand tracking are closer to the ground truth. The sign gesture recognition based on the proposed methods yields a 87.33% recognition rate for the American Sign Language. The proposed hand tracking and feature extraction methodology is an important milestone in the development of expert systems designed for sign language recognition, such as automated sign language translation systems.     

基于多传感器融合的动态手势识别研究分析

研究利用三类传感器（表面肌电仪、陀螺仪和加速度计）信号的特点进行信息融合,提高可识别动态手势动作的种类和准确率。将动态手势动作分解为手形、手势朝向和运动轨迹三个要素,分别使用表面肌电信号（sEMG）、陀螺仪信号（GYRO）和加速度信号（ACC）进行表征,利用多流HMMs进行动态手势动作的模式识别。对包含有5个运动轨迹和6个静态手形的识别实验结果表明,该方法可以有效地从连续信号中识别动态手势,三类传感器组合使用获得的全局平均识别率达到92％以上,明显高于任意两个传感器组合和仅采用单个传感器获得的平均识别率。实验表明该方法是一种有效的动态手势识别方法,并且相较于传统的动态手势识别的方法更具有优势。     

多尺度卷积特征融合的SSD手势识别算法

为了提高对中小占比手势识别的准确性与稳定性,提出了一种多尺度卷积特征融合的SSD(single shot multibox detector)手势识别方法。该方法突出表现在两大方面,其一,在原始的SSD算法的多尺度卷积检测方法基础上,引入了不同卷积层的特征融合思想,经过空洞卷积下采样操作与反卷积上采样操作,实现网络结构中的浅层视觉卷积层与深层语义卷积层的融合,代替原有的卷积层用于手势识别,以提高模型对中小目标手势的识别精度;其二,为了解决正负样本不均衡导致分类性能差的问题,提出一种改进的损失函数,以提升模型对目标手势的分类能力。在手势识别公开的数据集上的实验结果表明,与SSD和Faster R-CNN等识别方法相比,能够在保持较高的手势检测精度的同时,又具有较好的鲁棒性与检测速度。     

基于残差双注意力与跨级特征融合模块的静态手势识别

为解决卷积神经网络提取特征遗漏、手势多特征提取不充分问题,本文提出基于残差双注意力与跨级特征融合模块的静态手势识别方法.设计了一种残差双注意力模块,该模块对ResNet50网络提取的低层特征进行增强,能够有效学习关键信息并更新权重,提高对高层特征的注意力,然后由跨级特征融合模块对不同阶段的高低层特征进行融合,丰富高级特征图中不同层级之间的语义和位置信息,最后使用全连接层的Softmax分类器对手势图像进行分类识别.本文在ASL美国手语数据集上进行实验,平均准确率为99.68%,相比基础ResNet50网络准确率提升2.52%.结果验证本文方法能充分提取与复用手势特征,有效提高手势图像的识别精度.     

Hidden Markov model for human to computer interaction: a study on human hand gesture recognition

Human hand recognition plays an important role in a wide range of applications ranging from sign language translators, gesture recognition, augmented reality, surveillance and medical image processing to various Human Computer Interaction (HCI) domains. Human hand is a complex articulated object consisting of many connected parts and joints. Therefore, for applications that involve HCI one can find many challenges to establish a system with high detection and recognition accuracy for hand posture and/or gesture. Hand posture is defined as a static hand configuration without any movement involved. Meanwhile, hand gesture is a sequence of hand postures connected by continuous motions. During the past decades, many approaches have been presented for hand posture and/or gesture recognition. In this paper, we provide a survey on approaches which are based on Hidden Markov Models (HMM) for hand posture and gesture recognition for HCI applications.     

基于FPGA的实时手势识别系统

本文构建了一个基于FPGA的实时手势识别平台,并在该平台上实现了一种基于表面肌电(sEMG)信号和加速度(ACC)信号的手势识别算法。具体实现过程中,无线sEMG传感器和无线三轴ACC传感器穿戴于两手前臂实时获取sEMG信号和ACC信号,并以无线方式发送到数据处理模块。数据处理模块充分利用FPGA的并行处理优势,融合ACC和sEMG信息特征,实现了单双手手势的实时识别。经测试,本文所用的手势识别算法移植到FPGA中以后,识别速度明显提高,16个中国手语手势动作达到了95%以上的识别率。     

Visual interpretation of hand gestures for human-computerinteraction: a review

The use of hand gestures provides an attractive alternative to cumbersome interface devices for human-computer interaction (HCI). In particular, visual interpretation of hand gestures can help in achieving the ease and naturalness desired for HCI. This has motivated a very active research area concerned with computer vision-based analysis and interpretation of hand gestures. We survey the literature on visual interpretation of hand gestures in the context of its role in HCI. This discussion is organized on the basis of the method used for modeling, analyzing, and recognizing gestures. Important differences in the gesture interpretation approaches arise depending on whether a 3D model of the human hand or an image appearance model of the human hand is used. 3D hand models offer a way of more elaborate modeling of hand gestures but lead to computational hurdles that have not been overcome given the real-time requirements of HCI. Appearance-based models lead to computationally efficient “purposive” approaches that work well under constrained situations but seem to lack the generality desirable for HCI. We also discuss implemented gestural systems as well as other potential applications of vision-based gesture recognition. Although the current progress is encouraging, further theoretical as well as computational advances are needed before gestures can be widely used for HCI. We discuss directions of future research in gesture recognition, including its integration with other natural modes of human-computer interaction     

一种复合型手势识别方法研究

针对已有卷积神经网络在手势识别过程中精度不高的问题,提出了一种双通道卷积神经网络的特征融合与动态衰减学习率相结合的复合型手势识别方法。通过两个相互独立的通道进行手势图像的特征提取,首先使用SENet（Squeeze-and-Excitation Networks）构成的第一通道提取全局特征,然后使用RBNet（Residual Block Networks）构成的第二通道提取局部特征,并将全局特征和局部特征进行通道维度上的融合。同时,利用动态衰减的学习率训练双通道网络模型。与其他卷积神经网络模型的对比实验结果表明,提出的复合型手势识别方法的手势识别率高,参数数量少,适用于不同手势图像数据集的识别。     

基于DTW算法的肌电信号手势识别方法

为了提高肌电信号手势识别算法的准确度,增强实时性,提出了一种基于动态时间规整（DTW）算法的手势识别方法,该方法利用肌电信号（EMG）对个体间的手势进行识别。首先,采用滑动平均能量的方法对原始的EMG信号进行数据分割,探测有效动作;其次,对于分割的数据段使用平均绝对值（MAV）来提取信号特征;最后,用DTW算法将8维的EMG信号融合并计算测试样本和模版的相似度,其中采用了DTW算法寻找规整路径的方法进行了模板制作,实现了个体间的手势识别。实验结果表明,使用DTW算法对肌电信号进行手势识别,其动作识别的准确率达到96.09%,该方法计算速度快,实时性强。     

基于手势特征融合的操作动作识别

针对动态复杂场景下的操作动作识别,提出一种基于手势特征融合的动作识别框架,该框架主要包含RGB视频特征提取模块、手势特征提取模块与动作分类模块。其中RGB视频特征提取模块主要使用I3D网络提取RGB视频的时间和空间特征;手势特征提取模块利用Mask R-CNN网络提取操作者手势特征;动作分类模块融合上述特征,并输入到分类器中进行分类。在EPIC-Kitchens数据集上,提出的方法识别抓取手势的准确性高达89.63%,识别综合动作的准确度达到了74.67%。     

适用于机器人视觉的手势识别系统

手势是一种高效的人机交互和设备控制的方式,基于视觉的手势识别是人机交互、模式识别等领域的一个富有挑战性的研究课题。文章提出并实现了一个可用于与机器人交互的静态手势检测和识别系统。该系统用摇动检测的方法定位人手;用基于现场采样得到的肤色模型进行手的分割;用简化并改进的CAMSHIFT算法对手势进行跟踪;最后用模式识别的方法提取简单特征进行识别。实验证明,该系统快速、稳定而有效。     

基于圆弧扫描线的手势特征提取和实时手势识别

基于手势的人机交互是当前备受关注的自然人机交互模式之一,实时手势识别是其中最重要的步骤。本文提出了一种基于圆弧扫描线的手势特征提取和实时手势识别方法。首先,基于一种抽象描述手掌和五指关系的简洁人手海龟模型,结合肤色特征和腕部标记分割出人手部图像,并进行二值化处理 和统一尺寸来建立手势训练集。 然后,以手掌中心为圆心构造同心圆来提取训练集中不同手势样本的特征,并使用线性判别分析(Linear discriminant analysis,LDA)算法对手势特征向量进行离线预处理。最后,使用 改进的加权K近邻（Weighted-K-nearest neighbor,W-KNN）算法进行实时手势分类和识别。为了验证本文方法的有效性 ,在自建小型手势数据库上进行了算法分析和比较,并在多投影系统下进行实时交互测试。实验 结果表明本文算法具有较高的识别效率。     

基于Leap Motion的动态手势识别

动态手势识别作为人机交互的一个重要方向，在各个领域具有广泛的需求。相较于静态手势，动态手势的变化更为复杂，对其特征的充分提取与描述是准确识别动态手势的关键。为了解决对动态手势特征描述不充分的问题，利用高精度的Leap Motion传感器对手部三维坐标信息进行采集，提出了一种包含手指姿势和手掌位移的特征在内的、能够充分描述复杂动态手势的特征序列，并结合长短期记忆网络模型进行动态手势识别。实验结果表明，提出的方法在包含16种动态手势的数据集上的识别准确率为98.50%；与其他特征序列的对比实验表明，提出的特征序列，能更充分准确地描述动态手势特征。