Robotic grasping and manipulation through human visuomotor learning

A major goal of robotics research is to develop techniques that allow non-experts to teach robots dexterous skills. In this paper, we report our progress on the development of a framework which exploits human sensorimotor learning capability to address this aim. The idea is to place the human operator in the robot control loop where he/she can intuitively control the robot, and by practice, learn to perform the target task with the robot. Subsequently, by analyzing the robot control obtained by the human, it is possible to design a controller that allows the robot to autonomously perform the task. First, we introduce this framework with the ball-swapping task where a robot hand has to swap the position of the balls without dropping them, and present new analyses investigating the intrinsic dimension of the ball-swapping skill obtained through this framework. Then, we present new experiments toward obtaining an autonomous grasp controller on an anthropomorphic robot. In the experiments, the operator directly controls the (simulated) robot using visual feedback to achieve robust grasping with the robot. The data collected is then analyzed for inferring the grasping strategy discovered by the human operator. Finally, a method to generalize grasping actions using the collected data is presented, which allows the robot to autonomously generate grasping actions for different orientations of the target object.     

Robotic execution of everyday tasks by means of external vision/force control

In this article, we present an integrated manipulation framework for a service robot, that allows to interact with articulated objects at home environments through the coupling of vision and force modalities. We consider a robot which is observing simultaneously his hand and the object to manipulate, by using an external camera (i.e. robot head). Task-oriented grasping algorithms (Proc of IEEE Int Conf on robotics and automation, pp 1794–1799, 2007) are used in order to plan a suitable grasp on the object according to the task to perform. A new vision/force coupling approach (Int Conf on advanced robotics, 2007), based on external control, is used in order to, first, guide the robot hand towards the grasp position and, second, perform the task taking into account external forces. The coupling between these two complementary sensor modalities provides the robot with robustness against uncertainties in models and positioning. A position-based visual servoing control law has been designed in order to continuously align the robot hand with respect to the object that is being manipulated, independently of camera position. This allows to freely move the camera while the task is being executed and makes this approach amenable to be integrated in current humanoid robots without the need of hand-eye calibration. Experimental results on a real robot interacting with different kind of doors are presented.     

A new strategy combining empirical and analytical approaches for grasping unknown 3D objects

This paper proposes a novel strategy for grasping 3D unknown objects in accordance with their corresponding task. We define the handle or the natural grasping component of an object as the part chosen by humans to pick up this object. When humans reach out to grasp an object, it is generally in the aim of accomplishing a task. Thus, the chosen grasp is quite related to the object task. Our approach learns to identify object handles by imitating humans. In this paper, a new sufficient condition for computing force-closure grasps on the obtained handle is also proposed. Several experiments were conducted to test the ability of the algorithm to generalize to new objects. They also show the adaptability of our strategy to the hand kinematics.     

Dynamic grasp planning of multifingered robot hands based onasymptotic stability

We describe an approach for planning grasps of multifingered robot hands based on a small vibration model. Using features of the grasp configuration, we analyze asymptotic stability, contact situations, and uniaxial fingertip force constraints for the combined planning of finger posture and finger position, and characterize the generalized mass, damping, and stiffness. Choosing the largest time constant of the vibration model as an optimization criterion for planning finger postures and positions, the original problem of dynamic grasp planning is formulated as a nonlinear program. Simulation examples for a three-fingered robot hand grasping a spherical object demonstrate the effectiveness of the approach.     

Visual programming by demonstration of grasping skills in the context of a mobile service robot using 1D-topology based self-organizing-maps

An approach to the task of Programming by demonstration (PbD) of grasping skills is introduced, where a mobile service robot is taught by a human instructor how to grasp a specific object. In contrast to other approaches the instructor demonstrates the grasping action several times to the robot to increase reconstruction performance. Only the robot’s stereoscopic vision system is used to track the instructor’s hand. The developed tracking algorithm is designed to not need artificial markers, data gloves or being restricted to fixed or difficult to calibrate sensor installations while at the same time being real-time capable on a mobile service robot with limited resources. Due to the instructor’s repeated demonstrations and his low repeating accuracy, every time a grasp is demonstrated the instructor performs it differently. To compensate for these variations and also to compensate for tracking errors, the use of a Self-Organizing-Map (SOM) with a one-dimensional topology is proposed. This SOM is used to generalize over differently demonstrated grasping actions and to reconstruct the intended approach trajectory of the instructor’s hand while grasping an object. The approach is implemented and evaluated on the service robot TASER using synthetically generated data as well as real world data.     

A human-like real-time grasp synthesis method for humanoid robot hands

This paper addresses a real-time grasp synthesis of multi-fingered robot hands to find grasp configurations which satisfy the force closure condition of arbitrary shaped objects. We propose a fast and efficient grasp synthesis algorithm for planar polygonal objects, which yields the contact locations on a given polygonal object to obtain a force closure grasp by a multi-fingered robot hand. For an optimum grasp and real-time computation, we develop the preference and the hibernation process and assign the physical constraints of a humanoid hand to the motion of each finger. The preferences consist of each sublayer reflecting the primitive preference similar to the conditional behaviors of humans for given objectives and their arrangements are adjusted by the heuristics of human grasping. The proposed method reduces the computational time significantly at the sacrifice of global optimality, and enables grasp posture to be changeable within 2-finger and 3-finger grasp. The performance of the presented algorithm is evaluated via simulation studies to obtain the force-closure grasps of polygonal objects with fingertip grasps. The architecture suggested is verified through experimental implementation to our developed robot hand system by solving 2- or 3-finger grasp synthesis.     

Few-experiential learning system of robotic picking task with selective dual-arm grasping

Recently, robots are introduced to warehouses and factories for automation and are expected to execute dual-arm manipulation as human does and to manipulate large, heavy and unbalanced objects. We focus on target picking task in the cluttered environment and aim to realize a robot picking system which the robot selects and executes proper grasping motion from single-arm and dual-arm motion. In this paper, we propose a few-experiential learning-based target picking system with selective dual-arm grasping. In our system, a robot first learns grasping points and object semantic and instance label with automatically synthesized dataset. The robot then executes and collects grasp trial experiences in the real world and retrains the grasping point prediction model with the collected trial experiences. Finally, the robot evaluates candidate pairs of grasping object instance, strategy and points and selects to execute the optimal grasping motion. In the experiments, we evaluated our system by conducting target picking task experiments with a dual-arm humanoid robot Baxter in the cluttered environment as warehouse.     

Direction Kernels: using a simplified 3D model representation for grasping

Humans decide how to carry out a spontaneous interaction with an object by using the whole geometric information obtained from their eyes. The aim of this paper is to present how our object representation model MWS (Adán in Comput Vis Image Underst 79:281–307, 2000) can help a robot manipulator to make a single and reliable interaction. The contribution of this paper is particularly focused on the grasp synthesis stage. The main idea is that the grasping system, through MWS, can use non-strict-local features of the contact points to find a consistent grasping configuration. The Direction Kernels (DK) concept, which is integrated into the MWS model, is used to define a set of candidate contact-points and interaction regions. The set of DK is a global feature which represents the principal normal vectors of the object and their relative weight in a three-connectivity mesh model. Our method calculates the optimal grasp points (which are ordered according to the quality function) for two-finger grippers, whilst maintaining the requirements of force closure and safety of the grasp. Our strategy has been extensively tested on real free-shape objects using a 6 DOF industrial robot.     

虚拟手抓持规则融合策略研究

群体虚拟手抓持规则是虚拟手和虚拟物体进行抓持操作的交互规则,用于判定虚拟手是否能够成功抓持物体。对基于几何的虚拟手抓持规则和基于物理的虚拟手抓持规则分别进行了研究,针对基于几何的虚拟手抓持规则规则简单、仿真效果较差,基于物理模型的虚拟手抓持规则计算复杂、难以实现实时仿真的问题:（1）改进基于几何的虚拟手抓持规则,通过接触点位置、法矢和抓持面法矢制定抓持规则,使其效果逼近力封闭虚拟手抓持规则;（2）利用力封闭计算中抓持接触点和法矢不变的特性,通过内力配比避免了抓持操作中的非线性规划求解,使抓持操作阶段实现实时仿真;（3）通过几何约束进行初始抓持判断-力封闭计算校正-内力配比力封闭计算的策略,实现了完整的抓持过程实时仿真。设计的交互实验说明该抓持规则能实现高沉浸感和实时性的抓持仿真,可以应用到虚拟训练、虚拟装配等仿真平台。     

基于PVDF的三维力机器人触觉传感器的设计

当机器手抓取时物体受力信息检测是抓取过程顺利进行的基础,检测三维方向上的力便可充分反映物体的受力信息。当前用于抓取过程中三维力检测的触觉传感器还存在着一些不足,基于此,论文拟设计一种基于PVDF的三维力机器人触觉传感器。论文展示了传感器的结构设计,建立了压电薄膜及传感头结构的数学模型,设计了调理电路并对传感器进行测试和验证,结果表明该传感器能有效检测机器手抓取过程中的三维力信息。     

Pose Planning for a Mobile Manipulator Based on Joint Motions for Posture Adjustment to End-Effector Error

This paper proposes a motion planning method for a mobile manipulator. In general, humans can grasp an object by various ways which depend on object posture, position and so on. The objective of this paper is to present how to detect the pose of a mobile manipulator under the condition that several ways of grasping are given to the robot. Motion errors and object position errors are considered to detect robot pose in our method because these affect the grasp motion of the robot hand. Coping with these errors, we will propose an effective pose searching method for a mobile manipulator from numerous pose candidates. The performance of the proposed method is illustrated by simulation and experiment.     

Stability index and contact configuration planning for multifingered grasp

It is necessary to plan the contact configuration to guarantee a stable grasp. This article discusses the grasping stability of multifingered robot hands. The fingers are assumed to be point contacts with friction. A stability index for evaluating a grasp, which is proportional to the ellipsoidal volume in the grasping task space, is proposed. The invariance of the index is proved under an object linear coordinate transformation and under a change of the torque origin. The similar invariance of the index is also proved under a change of the dimensional unit. The optimal grasping of an object by a multifingered robot hand can be obtained using the stability index to plan the grasp configurations. The index is applicable to plan adaptable fixtures as well. A nonlinear programming method to plan configurations is addressed. Several examples are given using the index to evaluate a grasp, in which the obtained optimal grasping is consistent with what human beings expect. The sensibility of the optimal grasping is analyzed in these examples. © 1998 John Wiley & Sons, Inc.     

基于多约束条件的机器人抓取策略学习方法

针对机器人在多类别物体不同任务下的抓取决策问题,提出基于多约束条件的抓取策略学习方法.该方法以抓取对象特征和抓取任务属性为机器人抓取策略约束,通过映射人类抓取习惯规划抓取模式,并采用物体方向包围盒(OBB)建立机器人抓取规则,建立多约束条件的抓取模型.利用深度径向基(DRBF)网络模型结合减聚类算法(SCM)实现抓取策略的学习,两种算法的结合旨在提高学习鲁棒性与精确性.搭建以Refiex 1型灵巧手和AUBO六自由度机械臂组成的实验平台,对多类别物体进行抓取实验.实验结果表明,所提出方法使机器人有效学习到对多物体不同任务的最优抓取策略,具有良好的抓取决策能力.     

Combining active learning and reactive control for robot grasping

Grasping an object is a task that inherently needs to be treated in a hybrid fashion. The system must decide both where and how to grasp the object. While selecting where to grasp requires learning about the object as a whole, the execution only needs to reactively adapt to the context close to the grasp’s location. We propose a hierarchical controller that reflects the structure of these two sub-problems, and attempts to learn solutions that work for both. A hybrid architecture is employed by the controller to make use of various machine learning methods that can cope with the large amount of uncertainty inherent to the task. The controller’s upper level selects where to grasp the object using a reinforcement learner, while the lower level comprises an imitation learner and a vision-based reactive controller to determine appropriate grasping motions. The resulting system is able to quickly learn good grasps of a novel object in an unstructured environment, by executing smooth reaching motions and preshaping the hand depending on the object’s geometry. The system was evaluated both in simulation and on a real robot.     

Object handling control among two-wheel robots and a human operator: An empirical approach

This article presents object handling control between two-wheel robot manipulators, and a two-wheel robot and a human operator. The two-wheel robot has been built for serving humans in the indoor environment. It has two wheels to maintain balance and is able to make contact with a human operator via an object. A position-based impedance force control method is applied to maintain stable object-handling tasks. As the human operator pushes and pulls the object, the robot also reacts to maintain contact with the object by pulling and pushing against the object to regulate a specified force. Master and slave configuration of two-wheel robots is formed for handling an object, where the master robot or a human leads the slave robot equipped with a force sensor. Switching control from position to force or vice versa is presented. Experimental studies are performed to evaluate the feasibility of the object-handling task between two-wheel mobile robots, and the robot and a human operator.     

A strategy for fast grasping of unknown objects using partial shape information from range sensors

In this paper, we present a strategy for fast grasping of unknown objects based on the partial shape information from range sensors for a mobile robot with a parallel-jaw gripper. The proposed method can realize fast grasping of an unknown object without needing complete information of the object or learning from grasping experience. Information regarding the shape of the object is acquired by a 2D range sensor installed on the robot at an inclined angle to the ground. Features for determining the maximal contact area are extracted directly from the partial shape information of the unknown object to determine the candidate grasping points. Note that since the shape and mass are unknown before grasping, a successful and stable grasp cannot be in fact guaranteed. Thus, after performing a grasping trial, the mobile robot uses the 2D range sensor to judge whether the object can be lifted. If a grasping trial fails, the mobile robot will quickly find other candidate grasping points for another trial until a successful and stable grasp is realized. The proposed approach has been tested in experiments, which found that a mobile robot with a parallel-jaw gripper can successfully grasp a wide variety of objects using the proposed algorithm. The results illustrate the validity of the proposed algorithm in term of the grasping time.     

A HUMANLIKE GRASPING FORCE PLANNER FOR OBJECT MANIPULATION BY ROBOT MANIPULATORS

Recently robot manipulators have been expected to perform sophisticated tasks such as object manipulation, assembly tasks, or cooperative tasks with human workers. In order to realize these tasks with robot manipulators, it is important to understand the human strategy of object grasping and manipulation. In this study, we have examined how a human being decides the grasping force necessary to manipulate an unknown object in order to apply human object-grasping strategy for robotic systems. Experiments have been performed with several kinds of objects under several kinds of conditions to investigate how much grasping force human subjects generate. Adjustment strategy of human grasping force when the object is manipulated or in contact with an environment is also examined. Neural networks (the desired grasping force planner) that generate the humanlike desired grasping force are then designed for robotic systems. The effectiveness of the proposed desired grasping force planner is evaluated via experiments.     

多臂空间机器人稳定抓取力分配和柔顺控制策略

针对影响多臂抓取稳定性的接触力不平衡和接触振动问题,提出多臂空间机器人力分配和柔顺控制策略.首先,分析满足多臂稳定抓取的力学条件,基于摩擦锥约束设计抓取力安全系数,并将其引入力优化模型进行抓取力分配,实现目标物体稳定抓取条件下受力最小;然后,分析抓取过渡过程的振动成因,设计基于动能消耗的末端输出力控制策略实现快速振动抑制和柔顺抓取;最后,设计机械臂末端控制律切换策略,一旦在抓取过渡过程中发生接触脱离可引导其快速返回物体表面.仿真结果表明,所提出方法提升了稳定抓取安全裕度,显著降低了机械臂末端的振动幅值、持续时间和接触力,提升了空间机器人多臂抓取目标操作的稳定性和柔顺性.     

Demonstration-based learning and control for automatic grasping

We present a method for automatic grasp generation based on object shape primitives in a Programming by Demonstration framework. The system first recognizes the grasp performed by a demonstrator as well as the object it is applied on and then generates a suitable grasping strategy on the robot. We start by presenting how to model and learn grasps and map them to robot hands. We continue by performing dynamic simulation of the grasp execution with a focus on grasping objects whose pose is not perfectly known.     

Active learning of visual descriptors for grasping using non-parametric smoothed beta distributions

One of the basic skills for a robot autonomous grasping is to select the appropriate grasping point for an object. Several recent works have shown that it is possible to learn grasping points from different types of features extracted from a single image or from more complex 3D reconstructions. In the context of learning through experience, this is very convenient, since it does not require a full reconstruction of the object and implicitly incorporates kinematic constraints as the hand morphology. These learning strategies usually require a large set of labeled examples which can be expensive to obtain. In this paper, we address the problem of actively learning good grasping points to reduce the number of examples needed by the robot. The proposed algorithm computes the probability of successfully grasping an object at a given location represented by a feature vector. By autonomously exploring different feature values on different objects, the systems learn where to grasp each of the objects. The algorithm combines beta–binomial distributions and a non-parametric kernel approach to provide the full distribution for the probability of grasping. This information allows to perform an active exploration that efficiently learns good grasping points even among different objects. We tested our algorithm using a real humanoid robot that acquired the examples by experimenting directly on the objects and, therefore, it deals better with complex (anthropomorphic) hand–object interactions whose results are difficult to model, or predict. The results show a smooth generalization even in the presence of very few data as is often the case in learning through experience.