Interaction learning for dynamic movement primitives used in cooperative robotic tasks

Since several years dynamic movement primitives (DMPs) are more and more getting into the center of interest for flexible movement control in robotics. In this study we introduce sensory feedback together with a predictive learning mechanism which allows tightly coupled dual-agent systems to learn an adaptive, sensor-driven interaction based on DMPs. The coupled conventional (no-sensors, no learning) DMP-system automatically equilibrates and can still be solved analytically allowing us to derive conditions for stability. When adding adaptive sensor control we can show that both agents learn to cooperate. Simulations as well as real-robot experiments are shown. Interestingly, all these mechanisms are entirely based on low level interactions without any planning or cognitive component.     

Reinforcement learning to adjust parametrized motor primitives to new situations

Humans manage to adapt learned movements very quickly to new situations by generalizing learned behaviors from similar situations. In contrast, robots currently often need to re-learn the complete movement. In this paper, we propose a method that learns to generalize parametrized motor plans by adapting a small set of global parameters, called meta-parameters. We employ reinforcement learning to learn the required meta-parameters to deal with the current situation, described by states. We introduce an appropriate reinforcement learning algorithm based on a kernelized version of the reward-weighted regression. To show its feasibility, we evaluate this algorithm on a toy example and compare it to several previous approaches. Subsequently, we apply the approach to three robot tasks, i.e., the generalization of throwing movements in darts, of hitting movements in table tennis, and of throwing balls where the tasks are learned on several different real physical robots, i.e., a Barrett WAM, a BioRob, the JST-ICORP/SARCOS CBi and a Kuka KR?6.     

基于改进动态运动基元的6D轨迹规划

动态运动基元(DMPs)轨迹规划方法可以简化机械臂控制中参数调整的复杂过程,快速生成运动轨迹,但是面对姿态的流形特性以及跨零点情况,现有的DMPs很难达到预期的效果.本文提出了一种基于改进DMPs的笛卡尔空间6D轨迹规划方法.该方法采用四元数描述姿态,实现了位置轨迹与姿态轨迹的无奇异表示.通过解耦强迫函数与起–终点状态差值项之间的关联,消除了跨零点引起的轨迹抖动、无法生成与翻转等问题.此外,基于机械臂和障碍物间的距离与偏角建立了虚拟阻抗关系,并将其耦合到动力学模型中,实现了机械臂末端的避障控制,避免了避障行为过早问题,有利于减少消耗.机械臂6D轨迹规划仿真和实验表明,本文提出的改进DMPs方法有效.     

Analogue VLSI primitives for perceptual tasks in machine vision

A variety of computational tasks in early vision can be formulated through lattice networks. The cooperative action of these networks depends upon the topology of interconnections, both feedforward and recurrent ones. The Gabor-like impulse response of a 2nd-order lattice network (i.e. with nearest and next-to-nearest interconnections) is analysed in detail, pointing out how a near-optimal filtering behaviour in space and frequency domains can be achieved through excitatory/inhibitory interactions without impairing the stability of the system. These architectures can be mapped, very efficiently at transistor level, on VLSI structures operating as analogue perceptual engines. The hardware implementation of early vision tasks can, indeed, be tackled by combining these perceptual agents through suitable weighted sums. Various implementation strategies have been pursued with reference to: (i) the algorithm-circuit mapping (current-mode and transconductor approaches); (ii) the degree of programmability (fixed, selectable and tunable); and (iii) the implementation technology (2 and 0.8 gate lengths). Applications of the perceptual engine to machine vision algorithms are discussed.     

Reinforcement learning of multiple tasks using a hierarchical CMAC architecture

A reinforcement learning approach based on modular function approximation is presented. Cerebellar Model Articulation Controller (CMAC) networks are incorporated in the Hierarchical Mixtures of Experts (HME) architecture and the resulting architecture is referred to as HME-CMAC. A computationally efficient on-line learning algorithm based on the Expectation Maximization (EM) algorithm is proposed in order to achieve fast function approximation with the HME-CMAC architecture.

The Compositional Q-Learning (CQ-L) framework establishes the relationship between the Q-values of composite tasks and those of elemental tasks in its decomposition. This framework is extended here to allow rewards in non-terminal states. An implementation of the extended CQ-L framework using the HME-CMAC architecture is used to perform task decomposition in a realistic simulation of a two-linked manipulator having non-linear dynamics. The context-dependent reinforcement learning achieved by adopting this approach has advantages over monolithic approaches in terms of speed of learning, storage requirements and the ability to cope with changing goals.     

Cords: geometric curve primitives for modeling contact

Curves are perhaps the most versatile of modeling primitives in computer graphics. They define a rough structure for many surface-generation algorithms and are often fit to meaningful surface features for further shape modeling. Deformable objects such as hair and fur are simulated on finite element curve discretizations. Motion paths for planning and animation applications are tied to underlying curves. In this article we present a geometric curve primitive, known as a cord, which allows for interactive modeling of curves that contact complex geometry.     

Learning to pour with a robot arm combining goal and shape learning for dynamic movement primitives

When describing robot motion with dynamic movement primitives (DMPs), goal (trajectory endpoint), shape and temporal scaling parameters are used. In reinforcement learning with DMPs, usually goals and temporal scaling parameters are predefined and only the weights for shaping a DMP are learned. Many tasks, however, exist where the best goal position is not a priori known, requiring to learn it. Thus, here we specifically address the question of how to simultaneously combine goal and shape parameter learning. This is a difficult problem because learning of both parameters could easily interfere in a destructive way. We apply value function approximation techniques for goal learning and direct policy search methods for shape learning. Specifically, we use “policy improvement with path integrals” and “natural actor critic” for the policy search. We solve a learning-to-pour-liquid task in simulations as well as using a Pa10 robot arm. Results for learning from scratch, learning initialized by human demonstration, as well as for modifying the tool for the learned DMPs are presented. We observe that the combination of goal and shape learning is stable and robust within large parameter regimes. Learning converges quickly even in the presence of disturbances, which makes this combined method suitable for robotic applications.     

Trajectory generation and control of a knee exoskeleton based on dynamic movement primitives for sit-to-stand assistance

This paper presents a novel control approach for a knee exoskeleton to assist individuals with lower extremity weakness during sit-to-stand motion. The proposed method consists of a trajectory generator and an impedance controller. The trajectory generator uses a library of sample trajectories as the training data and the initial joint angles as the input to predict the user’s intended sit-to-stand trajectory. Utilizing the dynamic movement primitives theory, the trajectory generator represents the predicted trajectory in a time-normalized and rather a flexible framework. The impedance controller is then employed to provide assistance by guiding the knee joint to move along the predicted trajectory. Moreover, the human-exoskeleton interaction force is used as the feedback for on-line adaptation of the trajectory speed. The proposed control strategy was tested on a healthy adult who wore the knee exoskeleton on his leg. The subject was asked to perform a number of sit-to-stand movements from different sitting positions. Next, the measured data and the inverse dynamic model of the human-exoskeleton system are used to calculate the knee power and torque profiles. The results reveal that average muscle activity decreases when the subject is assisted by the exoskeleton.     

Probabilistic movement primitives under unknown system dynamics

Physical interaction requires robots to accurately follow kinematic trajectories while modulating the interaction forces to accomplish tasks and to be safe to the environment. However, current approaches rely on accurate physical models or iterative learning approaches. We present a versatile approach for physical interaction tasks, based on Movement Primitives (MPs) that can learn physical interaction tasks solely by demonstrations, without explicitly modeling the robot or the environment. We base our approach on the Probabilistic Movement Primitives (ProMPs), which utilizes the variance of the demonstrations to provide better generalization of the encoded skill, combine skills, and derive a controller that follows exactly the encoded trajectory distribution. However, the ProMP controller requires the system dynamics to be known. We present a reformulation of the ProMPs that allows accurate reproduction of the skill without modeling the system dynamics and, further, we extent our approach to incorporate external sensors, as for example, force/torque sensors. Our approach learns physical interaction tasks solely from demonstrations and online adapts the movement to force–torque sensor input. We derive a variable-stiffness controller in closed form that reproduces the trajectory distribution and the interaction forces present in the demonstrations. We evaluate our approach in simulated and real-robot tasks.     

基于强化学习的固定翼飞机姿态控制方法

研究基于强化学习的飞机姿态控制方法,控制器输入为飞机纵向和横向状态变量以及姿态误差,输出为升降舵和副翼偏转角度指令,实现不同初始条件下飞机姿态角快速响应,同时避免使用传统PID控制器和不同飞行状态下的参数调节.根据飞机姿态变换特性,通过设置分立的神经网络模型提高算法收敛效率.为贴近实际的固定翼飞机控制,仿真基于JSBSim的F-16飞机空气动力学模型,利用OpenAI gym搭建强化学习仿真环境,以任意角速度、角度和空速作为初始条件,对姿态控制器中的动作网络和评价网络进行训练.仿真结果表明,基于强化学习的姿态控制器响应速度快,动态误差小,并能避免大过载等边界条件.     

Integrating linguistic primitives in learning context-dependentrepresentation

The paper presents an explicit connectionist-inspired, language learning model in which the process of settling on a particular interpretation for a sentence emerges from the interaction of a set of “soft” lexical, semantic, and syntactic primitives. We address how these distinct linguistic primitives can be encoded from different modular knowledge sources but strongly involved in an interactive processing in such a way as to make implicit linguistic information explicit. The learning of a quasi-logical form called context-dependent representation, is inherently incremental and dynamical in such a way that every semantic interpretation will be related to what has already been presented in the context created by prior utterances. With the aid of the context-dependent representation, the capability of the language learning model in text understanding is strengthened. This approach also shows how the recursive and compositional role of a sentence as conveyed in the syntactic structure can be modeled in a neurobiologically motivated linguistics based on dynamical systems rather on combinatorial symbolic architecture. Experiments with more than 2000 sentences in different languages illustrating the influences of the context-dependent representation on semantic interpretation, among other issues, are included     

From dynamic movement primitives to associative skill memories

In recent years, research on movement primitives has gained increasing popularity. The original goals of movement primitives are based on the desire to have a sufficiently rich and abstract representation for movement generation, which allows for efficient teaching, trial-and-error learning, and generalization of motor skills (Schaal 1999). Thus, motor skills in robots should be acquired in a natural dialog with humans, e.g., by imitation learning and shaping, while skill refinement and generalization should be accomplished autonomously by the robot. Such a scenario resembles the way we teach children and connects to the bigger question of how the human brain accomplishes skill learning. In this paper, we review how a particular computational approach to movement primitives, called dynamic movement primitives, can contribute to learning motor skills. We will address imitation learning, generalization, trial-and-error learning by reinforcement learning, movement recognition, and control based on movement primitives. But we also want to go beyond the standard goals of movement primitives. The stereotypical movement generation with movement primitives entails predicting of sensory events in the environment. Indeed, all the sensory events associated with a movement primitive form an associative skill memory that has the potential of forming a most powerful representation of a complete motor skill.     

Deep learning based robot cognitive architecture for collaborative assembly tasks

As the manufacturing industry becomes more agile, the use of collaborative robots capable of safely working with humans is becoming more prevalent, while adaptable and natural interaction is a goal yet to be achieved. This work presents a cognitive architecture composed of perception and reasoning modules that allows a robot to adapt its actions while collaborating with humans in an assembly task. Human action recognition perception is performed using convolutional neural network models with inertial measurement unit and skeleton tracking data. The action predictions are used for task status reasoning which predicts the time left for each action in a task allowing a robot to plan future actions. The task status reasoning uses a recurrent neural network method which is developed for transferability to new actions and tasks. Updateable input parameters allowing the system to optimise for each user and task with each trial performed are also investigated. Finally, the complete system is demonstrated with the collaborative assembly of a small chair and wooden box, along with a solo robot task of stacking objects performed when it would otherwise be idle. The human actions recognised are using a screw driver, Allen key, hammer and hand screwing, with online accuracies between 83–92%. User trials demonstrate the robot deciding when to start collaborative actions in order to synchronise with the user, as well as deciding when it has time to complete an action on its solo task before a collaborative action is required.     

手写运动的协作基元合成分析方法

研究一种生物运动神经控制机理与数据合成分析相结合的手写运动分析方法.特别地,将运动协作基元的概念用于手写运动数据分析,研究手写运动的协作基元合成分析方法,建立符合生物运动神经控制规律的手写运动数据理解模式.提出的协作基元合成分析过程由2个交替迭代的优化算法组成:其一,基于非负矩阵因子分解模式估计协作基元及调制幅度;其二,采用相似性最大化准则估计协作基元的激活时间.针对笔画切分的实验研究表明,采用协作基元合成分析方法获得的笔画切分结果,能够很好揭示相邻笔画之间的重叠连接模式,证实了所提方法的有效性.     

Reinforcement learning system based on heuristics free state focusing

We discuss the role of state focus in reinforcement learning (RL) systems that are applicable to mechanical systems including robots. Although the concept of the state focus is similar to attention/focusing in visual domains, its implementation requires some theoretical background based on RL. We propose an RL system that effectively learns how to choose the focus simultaneously with how to achieve a task. This RL system does not need heuristics for the adaptation of its focus. We conducted a capture experiment to compare the learning speed between the proposed system and the traditional systems, SARSAs, and conducted a navigation experiment to confirm the applicability of the proposed system to a realistic task. In the capture experiment, the proposed system learned faster than SARSAs. We visualized the developmental process of the focusing strategy in the proposed system using a Q-value analysis technique. In the navigation task, the proposed system demonstrated faster learning than SARSAs in the realistic task. The proposed system is applicable to a wide class of RLs that are applicable to mechanical systems including robots.     

Automata for learning sequential tasks

This paper describes a system that is capable of learning both combinational and sequential tasks. The system learns from sequences of input/output examples in which each pair of input and output represents a step in a task. The system uses finite state machines as its internal models. This paper proposes a method for inferring finite state machines from examples. New algorithms are developed for modifying the finite state machines to allow the system to adapt to changes. In addition, new algorithms are developed to allow the system to handle inconsistent information that may result from noise in the training examples. The system can handle sequential tasks involving long-term dependencies for which recurrent neural networks have been shown to be inadequate. Moreover, the learned finite state machines are easy to be implemented in VLSI. The system has a wide range of applications including but not limited to (a) sequence detection, prediction, and production, (b) intelligent macro systems that can learn rather than simply record sequences of steps performed by a computer user, and (c) design automation systems for designing finite state machines or sequential circuits. C. H. Ben CHOI, Ph.D.: He got his B. S., M. S., and Ph.D. degrees all from The Ohio State University in the United States. His major areas of study include Solid State Microelectronics, Computer Engineering, and Computer Information Science. He has works on general associative memory, parallel and distributed computer architectures, and machine learning. He currently works on a project concerning theoretical aspects of learning machine. His research interests include hardware and software methods of building an intelligent machine.     

Adaptive control of robot contact tasks with on-line learning of planar surfaces

In robot constrained motion problems on planar surfaces with frictional contacts, uncertainties on the contacted surface not only affect the control system performance but also distort control targets. The surface normal direction cosines are in this case uncertain parameters that are involved in both the control law and the control targets. This work proposes an adaptive learning controller that uses force and joint position/velocity measurements to simultaneously learn the surface orientation and achieve the desired goal. Simulation examples for a 6 dof robot are used to illustrate the theoretical results and the performance of the proposed controller in practical cases.     

Learning task-parameterized dynamic movement primitives using mixture of GMMs

Task-parameterized skill learning aims at adaptive motion encoding to new situations. While existing approaches for task-parameterized skill learning have demonstrated good adaptation within the demonstrated region, the extrapolation problem of task-parameterized skills has not been investigated enough. In this work, with the aim of good adaptation not only within the demonstrated region but also outside of the region, we propose to combine a generative model with a dynamic movement primitive by formulating learning as a density estimation problem. Moreover, for efficient learning from relatively few demonstrations, we propose to augment training data with additional incomplete data. The proposed method is tested and compared with existing works in simulations and real robot experiments. Experimental results verified its generalization in the extrapolation region.     

基于原语的通用起源过滤框架

现有起源过滤机制的通用性差,一个过滤机制仅能过滤某一特定类型的敏感元素,处理包含多种类型敏感元素的综合性起源过滤需求仍然非常困难,为此提出了一种基于原语的通用起源过滤框架。首先,介绍了起源过滤涉及的敏感元素类型以及过滤约束;其次,深入分析已有过滤机制改造起源图的基本操作和过程,形式地定义了一系列起源过滤原语,描述针对起源图的最小改造操作,将起源过滤过程划分为隐藏敏感元素、恢复有用依赖和验证过滤约束三个阶段,提出了一种基于原语组装的分阶段过滤策略空间构造方法;最后设计并实现了基于原语的通用过滤算法,并在公开数据集上验证了该算法的可行性。     

Hierarchical mesh segmentation based on fitting primitives

In this paper, we describe a hierarchical face clustering algorithm for triangle meshes based on fitting primitives belonging to an arbitrary set. The method proposed is completely automatic, and generates a binary tree of clusters, each of which is fitted by one of the primitives employed. Initially, each triangle represents a single cluster; at every iteration, all the pairs of adjacent clusters are considered, and the one that can be better approximated by one of the primitives forms a new single cluster. The approximation error is evaluated using the same metric for all the primitives, so that it makes sense to choose which is the most suitable primitive to approximate the set of triangles in a cluster. Based on this approach, we have implemented a prototype that uses planes, spheres and cylinders, and have experimented that for meshes made of 100 K faces, the whole binary tree of clusters can be built in about 8 s on a standard PC. The framework described here has natural application in reverse engineering processes, but it has also been tested for surface denoising, feature recovery and character skinning.