Deep learning-based human action recognition to leverage context awareness in collaborative assembly

Human–Robot Collaboration is a critical component of Industry 4.0, contributing to a transition towards more flexible production systems that are quickly adjustable to changing production requirements. This paper aims to increase the natural collaboration level of a robotic engine assembly station by proposing a cognitive system powered by computer vision and deep learning to interpret implicit communication cues of the operator. The proposed system, which is based on a residual convolutional neural network with 34 layers and a long-short term memory recurrent neural network (ResNet-34 + LSTM), obtains assembly context through action recognition of the tasks performed by the operator. The assembly context was then integrated in a collaborative assembly plan capable of autonomously commanding the robot tasks. The proposed model showed a great performance, achieving an accuracy of 96.65% and a temporal mean intersection over union (mIoU) of 94.11% for the action recognition of the considered assembly. Moreover, a task-oriented evaluation showed that the proposed cognitive system was able to leverage the performed human action recognition to command the adequate robot actions with near-perfect accuracy. As such, the proposed system was considered as successful at increasing the natural collaboration level of the considered assembly station.     

A decision model on human-robot collaborative routing for automatic logistics

This paper studies an integrated human-robot routing problem arising in collaborative order picking systems. Travelling routes of pickers and robots interact with each other and simultaneously determine the start time of tasks. A mixed-integer programming model is proposed which aims to minimize the total task completion time by optimizing both pickers’ and robots’ routes. Moreover, a variable neighborhood search and dynamic programming -based heuristic algorithm is developed for solving the problem in large-scale environments. Numerical experiments are conducted to prove the necessity of the proposed model and also validate the efficiency of the proposed heuristic algorithm. The results indicate the methodology can yield solutions with smaller optimality gap within a reasonable period of time, in comparison with an intuitive but common decision rule. A sensitivity analysis is also conducted to derive a number of managerial implications regarding the configuration of pickers and robots and robots’ speed setting.     

Human motion prediction for human-robot collaboration

In human-robot collaborative manufacturing, industrial robots would work alongside human workers who jointly perform the assigned tasks seamlessly. A human-robot collaborative manufacturing system is more customised and flexible than conventional manufacturing systems. In the area of assembly, a practical human-robot collaborative assembly system should be able to predict a human worker’s intention and assist human during assembly operations. In response to the requirement, this research proposes a new human-robot collaborative system design. The primary focus of the paper is to model product assembly tasks as a sequence of human motions. Existing human motion recognition techniques are applied to recognise the human motions. Hidden Markov model is used in the motion sequence to generate a motion transition probability matrix. Based on the result, human motion prediction becomes possible. The predicted human motions are evaluated and applied in task-level human-robot collaborative assembly.     

A digital twin-driven human-robot collaborative assembly approach in the wake of COVID-19

In the wake of COVID-19, the production demand of medical equipment is increasing rapidly. This type of products is mainly assembled by hand or fixed program with complex and flexible structure. However, the low efficiency and adaptability in current assembly mode are unable to meet the assembly requirements. So in this paper, a new framework of human-robot collaborative (HRC) assembly based on digital twin (DT) is proposed. The data management system of proposed framework integrates all kinds of data from digital twin spaces. In order to obtain the HRC strategy and action sequence in dynamic environment, the double deep deterministic policy gradient (D-DDPG) is applied as optimization model in DT. During assembly, the performance model is adopted to evaluate the quality of resilience assembly. The proposed framework is finally validated by an alternator assembly case, which proves that DT-based HRC assembly has a significant effect on improving assembly efficiency and safety.     

面向对手建模的意图识别方法综述

首先介绍了对手建模的几种不同的类型,引出行为建模中的意图识别问题;随后针对意图识别的过程、分类、主要研究方法、研究展望以及实际应用进行了归纳分析,总结并讨论了相关领域取得的最新研究成果;最后指出意图识别目前存在的不足以及未来的发展方向.     

Ontology-based state representations for intention recognition in human–robot collaborative environments

In this paper, we describe a novel approach for representing state information for the purpose of intention recognition in cooperative human–robot environments. States are represented by a combination of spatial relationships in a Cartesian frame along with cardinal direction information. This approach is applied to a manufacturing kitting operation, where humans and robots are working together to develop kits. Based upon a set of predefined high-level state relationships that must be true for future actions to occur, a robot can use the detailed state information described in this paper to infer the probability of subsequent actions occurring. This would allow the robot to better help the human with the task or, at a minimum, better stay out of his or her way.     

Probabilistic knowledge infusion through symbolic features for context-aware activity recognition

In the general machine learning domain, solutions based on the integration of deep learning models with knowledge-based approaches are emerging. Indeed, such hybrid systems have the advantage of improving the recognition rate and the model’s interpretability. At the same time, they require a significantly reduced amount of labeled data to reliably train the model. However, these techniques have been poorly explored in the sensor-based Human Activity Recognition (HAR) domain. The common-sense knowledge about activity execution can potentially improve purely data-driven approaches. While a few knowledge infusion approaches have been proposed for HAR, they rely on rigid logic formalisms that do not take into account uncertainty. In this paper, we propose P-NIMBUS, a novel knowledge infusion approach for sensor-based HAR that relies on probabilistic reasoning. A probabilistic ontology is in charge of computing symbolic features that are combined with the features automatically extracted by a CNN model from raw sensor data and high-level context data. In particular, the symbolic features encode probabilistic common-sense knowledge about the activities consistent with the user’s surrounding context. These features are infused within the model before the classification layer. We experimentally evaluated P-NIMBUS on a HAR dataset of mobile devices sensor data that includes 14 different activities performed by 25 users. Our results show that P-NIMBUS outperforms state-of-the-art neuro-symbolic approaches, with the advantage of requiring a limited amount of training data to reach satisfying recognition rates (i.e., more than 80% of F1-score with only 20% of labeled data).     

Soft computing based intention reading techniques as a means of human-robot interaction for human centered system

 Friendly interaction between robot and human is vital in the design of a human centered system. Among several interaction technologies, the intention reading of the user plays an important role for the human centered system. We focus on the aspect of intention reading in rehabilitation robots, and implement its capability for the wheelchair-based robotic arm system, called KARES (KAIST Rehabilitation Engineering Service System) II. An effective intention reading scheme is proposed on the basis of several soft computing techniques to handle uncertainty of the user's intention. Two application examples of intention reading in KARES II are discussed: one is visual servoing of the user's face, and the other is emergency stop of the robot by using EMG signals of the user's arm. This work is partially supported by the Ministry of Science and Technology of Korea as a part of Critical Technology 21 Program on “Development of Service Robot Technology”.     

基于B/S结构的协同装配的关键技术

分析了协同设计中协同装配的应用模式。在具备协同感知的前提下针对同步协同装配和异步协同装配两种应用模式分别提出对应的体系结构设计。提出基于B／S结构协同装配的具体实现方法，并实现了一套基于B／S结构的协同装配工具。     

基于DenseNet的无人汽车制动意图识别方法

无人汽车制动意图内部数据由于识别深度增加,会出现过度膨胀现象,导致制动意图数据收集完整度低、识别准确率差。提出基于DenseNet的无人汽车制动示意图识别方法。选择数据深度收集系统,收集无人汽车制动意图内部数据,结合电池保护模型深度分解汽车内部运行过程的能耗,以收集的初始内部数据为标准,整合无人汽车制动意图识别数据,拆分整合数据,防止数据过度膨胀。利用DenseNet的高学习度以及自适应学习性,加权均衡处理内部数据标定函数,设置一组基函数,并选择相应的DenseNet复制内部数据函数,自适应分析复制后的数据,完成制动意图识别。实验结果表明,制动意图数据收集完整度提高15.21%,识别准确率增强了23.68%。     

改进高维数据相似度的目标意图识别方法

对于实际战场中目标属性要素呈现出的多样化,传统目标意图识别方法不能够较全面地建立属性之间的相似度模型.为了更好地阐述实际战场的复杂情况,提高目标意图识别的准确度,提出了一种利用改进的空间相似度与属性相似度融合的高维数据相似度模型,以全面地计算目标各种属性状态对态势意图的支持程度,再利用得到的高维数据相似度通过D-S证据理论对目标进行序贯识别.仿真实验表明:该方法具有有效性以及能够提高目标意图识别的准确度,为解决目标战术意图识别提供了新的方法.     

Deep learning based robot cognitive architecture for collaborative assembly tasks

As the manufacturing industry becomes more agile, the use of collaborative robots capable of safely working with humans is becoming more prevalent, while adaptable and natural interaction is a goal yet to be achieved. This work presents a cognitive architecture composed of perception and reasoning modules that allows a robot to adapt its actions while collaborating with humans in an assembly task. Human action recognition perception is performed using convolutional neural network models with inertial measurement unit and skeleton tracking data. The action predictions are used for task status reasoning which predicts the time left for each action in a task allowing a robot to plan future actions. The task status reasoning uses a recurrent neural network method which is developed for transferability to new actions and tasks. Updateable input parameters allowing the system to optimise for each user and task with each trial performed are also investigated. Finally, the complete system is demonstrated with the collaborative assembly of a small chair and wooden box, along with a solo robot task of stacking objects performed when it would otherwise be idle. The human actions recognised are using a screw driver, Allen key, hammer and hand screwing, with online accuracies between 83–92%. User trials demonstrate the robot deciding when to start collaborative actions in order to synchronise with the user, as well as deciding when it has time to complete an action on its solo task before a collaborative action is required.     

基于局部结构的多尺度协作表示人脸识别算法

人脸识别在实际应用中,往往存在无法获取足够多的训练样本的情况,而在小样本情况下,协作表示的识别性能会受到严重影响。多尺度块协作表示算法能有效集成不同尺度下的分类结果,但其分类框架中子块的计算是相互独立的,忽略了块之间的结构关系。而局部结构法将图像划分为多个局部区域,每个局部区域的重叠块分布在相同的线性子空间中,该子空间可以反应块之间的结构关系,能提高多尺度块协作表示在小样本下的鲁棒性。因此提出了基于局部结构的多尺度块协同表示算法（Local Structure based Multi-Patch Collaborative Representation,LS_MPCRC）,在Yale B和AR人脸库上的实验结果证明,该算法在训练样本数目较少时具有优秀的识别性能。     

基于低秩矩阵恢复与协同表征的人脸识别算法

针对人脸图像不完备的问题和人脸图像在不同视角、光照和噪声下所造成训练样本污损的问题,提出了一种快速的人脸识别算法--RPCA_CRC。首先,将人脸训练样本对应的矩阵D₀分解为类间低秩矩阵D和稀疏误差矩阵E;其次,以低秩矩阵D为基础,得到测试样本的协同表征;最后,通过重构误差进行分类。相对于基于稀疏表征的分类(SRC)方法,所提算法运行速度平均提高25倍;且在训练样本数不完备的情况下,识别率平均提升30%。实验证明该算法快速有效,识别率高。     

区分交互意图的图卷积协同过滤算法

近几年提出了一些基于图卷积网络的协同过滤推荐模型,然而大部分模型将邻域权重视为常量且不区分用户和物品间的交互关系,无法获取令用户满意的推荐列表。因此,为了得到用户和物品更准确的嵌入表示,提出一种区分交互意图的图卷积协同过滤推荐算法MiGCCF(multi-intention graph convolutional collaborative filtering)。该算法将交互关系进行分解,细粒度分析用户与物品间的交互意图,并引入注意力机制,在消息传播过程中赋予邻域可学习的注意力权重,挖掘用户对于不同交互物品的喜爱度。在Gowalla与Amazon-book上的实验表明,该算法相比于基准算法,在两个数据集上的HR@50和NDCG@50指标分别提高了12.5%和8.5%,具有更好的性能表现。     

Design formalism for collaborative assembly design

Joints in product design are common because of the limitations of component geometric configurations and material properties, and the requirements of inspection, accessibility, repair, and portability. Collaborative product design is emerging as a viable alternative to the traditional design process. The collaborative assembly design (AsD) methodologies are needed for distributed product development. Existing AsD methodologies have limitations in capturing the non-geometric aspects of designer's intent on joining and are not efficient for a collaborative design environment. This paper introduces an AsD formalism and associated AsD tools to capture joining relations and spatial relationship implications. This AsD formalism allows the joining relations to be modeled symbolically for computer interpretation, and the model can be used for inferring mathematical and physical implications. An AsD model generated from the AsD formalism is used to exchange AsD information transparently in a collaborative AsD environment. An assembly relation model and a generic assembly relationship diagram are to capture assembly and joining information concisely and persistently. As a demonstration, the developed AsD formalism and AsD tools are applied on a connector assembly with arc weld and rivet joints.     

Geometric and form feature recognition tools applied to a design for assembly methodology

The paper presents geometric tools for an automated Design for Assembly (DFA) assessment system. For each component in an assembly a two step features search is performed: firstly (using the minimal bounding box) mass, dimensions and symmetries are identified allowing the part to be classified, according to DFA convention, as either rotational or prismatic; secondly form features are extracted allowing an effective method of mechanised orientation to be determined. Together these algorithms support the fuzzy decision support system, of an assembly-orientated CAD system known as FuzzyDFA.     

Driving intention recognition and behaviour prediction based on a double-layer hidden Markov model

We propose a model structure with a double-layer hidden Markov model (HMM) to recognise driving intention and predict driving behaviour. The upper-layer multi-dimensional discrete HMM (MDHMM) in the double-layer HMM represents driving intention in a combined working case, constructed according to the driving behaviours in certain single working cases in the lower-layer multi-dimensional Gaussian HMM (MGHMM). The driving behaviours are recognised by manoeuvring the signals of the driver and vehicle state information, and the recognised results are sent to the upper-layer HMM to recognise driving intentions. Also, driving behaviours in the near future are predicted using the likelihood-maximum method. A real-time driving simulator test on the combined working cases showed that the double-layer HMM can recognise driving intention and predict driving behaviour accurately and efficiently. As a result, the model provides the basis for pre-warning and intervention of danger and improving comfort performance.     

融合局部思想和协作表达的鲁棒分类

目的 协作表达分类算法在人脸识别实验上表现出较好的性能,但其未考虑样本的局部特性,且算法只能处理测试样本中的噪声,未能有效处理训练样本集中的噪声.针对这两个问题,提出融合局部思想和协作表达的鲁棒分类算法.方法 一方面,在训练集上,通过奇异值分解SVD得到其有效表达,丢弃一些噪声;另一方面,算法考虑数据的局部相似性,以保持测试样本与其相邻训练样本之间的相似性.结果 本文算法能得到一个闭式(closed-form),可避免稀疏表示分类算法中由于迭代引起的高时间复杂度问题,在ORL、扩展YALEB和PIE人脸库上的识别率分别可达91.4%,93.8%和93.2%,与同类算法相比识别率有较大幅度地提高;实验结果验证了算法所得到的系数具有较高的判别能力.结论 算法将训练样本进行奇异值分解得到“干净”的训练样本,能在一定程度上消除噪声的影响,且在协作表达的基础上,考虑测试样本和与之相邻的训练样本的局部相似性,相比原始的协作表达分类算法有更好的稳定性和鲁棒性.     

Ontology-based assembly design and information sharing for collaborative product development

To realize a truly collaborative product design and development process, effective communication among design collaborators is a must. In other words, the design intent that is imposed in a product design should be seized and interpreted properly; heterogeneous modeling terms should be semantically processed both by design collaborators and intelligent systems. Ontologies in the Semantic Web can explicitly represent semantics and promote integrated and consistent access to data and services. Thus, if an ontology is used in a heterogeneous and distributed design collaboration, it will explicitly and persistently represent engineering relations that are imposed in an assembly design. Design intent can be captured by reasoning, and, in turn, as reasoned facts, it can be propagated and shared with design collaborators. This paper presents a new paradigm of ontology-based assembly design. In the framework, an assembly design (AsD) ontology serves as a formal, explicit specification of assembly design so that it makes assembly knowledge both machine-interpretable and to be shared. An Assembly Relation Model (ARM) is enhanced using ontologies that represent engineering, spatial, assembly, and joining relations of assembly in a way that promotes collaborative assembly information-sharing environments. In the developed AsD ontology, implicit AsD constraints are explicitly represented using OWL (Web Ontology Language) and SWRL (Semantic Web Rule Language). This paper shows that the ability of the AsD ontology to be reasoned can capture both assembly and joining intents by a demonstration with a realistic mechanical assembly. Finally, this paper presents a new assembly design information-sharing framework and an assembly design browser for a collaborative product development.