Robotic grasping: from wrench space heuristics to deep learning policies

The robotic grasping task persists as a modern industry problem that seeks autonomous, fast implementation, and efficient techniques. Domestic robots are also a reality demanding a delicate and accurate human–machine interaction, with precise robotic grasping and handling. From decades ago, with analytical heuristics, to recent days, with the new deep learning policies, grasping in complex scenarios is still the aim of several works’ that propose distinctive approaches. In this context, this paper aims to cover recent methodologies’ development and discuss them, showing state-of-the-art challenges and the gap to industrial applications deployment. Given the complexity of the related issue associated with the elaborated proposed methods, this paper formulates some fair and transparent definitions for results’ assessment to provide researchers with a clear and standardised idea of the comparison between the new proposals.     

Robotic grasp manipulation using evolutionary computing and deep reinforcement learning

Intelligent Service Robotics - Intelligent object manipulation for grasping is a challenging problem for robots. Unlike robots, humans almost immediately know how to manipulate objects for grasping...     

Two-arm manipulation tasks with friction-assisted grasping

—The main objective of this paper is to study human dual-arm manipulation tasks and to develop a computational model that predicts the trajectories and force distribution for the coordination of two arms moving an object between two given positions and orientations in a horizontal plane. Our ultimate goal is to understand the dynamics of human dual-arm coordination in order to develop better robot control algorithms. We propose a computational model based on the hypothesis proposed by Uno et al. that suggests that human movements minimize the integral of the norm of the rate of change of actuator torques. We compare the experimental trajectories and force distributions with those obtained from the computational model. The observed trajectories show a significant degree of repeatability across trials and across subjects. We show that the computational model predicts the trajectories and the distribution of forces (torques) for a certain class of trajectories. However, the trajectories in the sagittal and frontal plane are characterized by asymmetric features that are hard to model using any integral cost function. Finally, we show that the computational model can be used to generate smooth trajectories and actuator forces for cooperating robots and discuss the advantages of such an approach to motion planning.     

Improving human robot collaboration through Force/Torque based learning for object manipulation

Human–Robot Collaboration (HRC) is a term used to describe tasks in which robots and humans work together to achieve a goal. Unlike traditional industrial robots, collaborative robots need to be adaptive; able to alter their approach to better suit the situation and the needs of the human partner. As traditional programming techniques can struggle with the complexity required, an emerging approach is to learn a skill by observing human demonstration and imitating the motions; commonly known as Learning from Demonstration (LfD). In this work, we present a LfD methodology that combines an ensemble machine learning algorithm (i.e. Random Forest (RF)) with stochastic regression, using haptic information captured from human demonstration. The capabilities of the proposed method are evaluated using two collaborative tasks; co-manipulation of an object (where the human provides the guidance but the robot handles the objects weight) and collaborative assembly of simple interlocking parts. The proposed method is shown to be capable of imitation learning; interpreting human actions and producing equivalent robot motion across a diverse range of initial and final conditions. After verifying that ensemble machine learning can be utilised for real robotics problems, we propose a further extension utilising Weighted Random Forest (WRF) that attaches weights to each tree based on its performance. It is then shown that the WRF approach outperforms RF in HRC tasks.     

Extracting data from human manipulation of objects towards improving autonomous robotic grasping

Humans excel in manipulation tasks, a basic skill for our survival and a key feature in our manmade world of artefacts and devices. In this work, we study how humans manipulate simple daily objects, and construct a probabilistic representation model for the tasks and objects useful for autonomous grasping and manipulation by robotic hands. Human demonstrations of predefined object manipulation tasks are recorded from both the human hand and object points of view. The multimodal data acquisition system records human gaze, hand and fingers 6D pose, finger flexure, tactile forces distributed on the inside of the hand, colour images and stereo depth map, and also object 6D pose and object tactile forces using instrumented objects. From the acquired data, relevant features are detected concerning motion patterns, tactile forces and hand-object states. This will enable modelling a class of tasks from sets of repeated demonstrations of the same task, so that a generalised probabilistic representation is derived to be used for task planning in artificial systems. An object centred probabilistic volumetric model is proposed to fuse the multimodal data and map contact regions, gaze, and tactile forces during stable grasps. This model is refined by segmenting the volume into components approximated by superquadrics, and overlaying the contact points used taking into account the task context. Results show that the features extracted are sufficient to distinguish key patterns that characterise each stage of the manipulation tasks, ranging from simple object displacement, where the same grasp is employed during manipulation (homogeneous manipulation) to more complex interactions such as object reorientation, fine positioning, and sequential in-hand rotation (dexterous manipulation). The framework presented retains the relevant data from human demonstrations, concerning both the manipulation and object characteristics, to be used by future grasp planning in artificial systems performing autonomous grasping.     

Slippage control in soft finger grasping and manipulation

Handling objects with robotic soft fingers without considering the odds of slippage are not realistic. Grasping and manipulation algorithms have to be tested under such conditions for evaluating their robustness. In this paper, a dynamic analysis of rigid object manipulation with slippage control is studied using a two-link finger with soft hemispherical tip. Dependency on contact forces applied by a soft finger while grasping a rigid object is examined experimentally. A power-law model combined with a linear viscous damper is used to model the elastic behavior and damping effect of the soft tip, respectively. In order to obtain precise dynamic equations governing the system, two second-order differential equations with variable coefficients have been designed to describe the different possible states of the contact forces accordingly. A controller is designed based on the rigid fingertip model using the concept of feedback linearization for each phase of the system dynamics. Numerical simulations are used to evaluate the performance of the controller. The results reveal that the designed controller shows acceptable performance for both soft and rigid finger manipulation in reducing and canceling slippage. Furthermore, simulations indicate that the applied force in the soft finger manipulation is considerably less than the rigid “one.”.     

Continuous control actions learning and adaptation for robotic manipulation through reinforcement learning

Autonomous Robots - This paper presents a learning-based method that uses simulation data to learn an object manipulation task using two model-free reinforcement learning (RL) algorithms. The...     

Robust object manipulation for tactile-based blind grasping

Tactile-based blind grasping addresses realistic robotic grasping in which the hand only has access to proprioceptive and tactile sensors. The robotic hand has no prior knowledge of the object/grasp properties, such as object weight, inertia, and shape. There exists no manipulation controller that rigorously guarantees object manipulation in such a setting. Here, a robust control law is proposed for object manipulation in tactile-based blind grasping. The analysis ensures semi-global asymptotic and exponential stability in the presence of model uncertainties and external disturbances that are neglected in related work. Simulation and hardware results validate the effectiveness of the proposed approach.     

Hierarchical reinforcement learning of multiple grasping strategies with human instructions

Grasping is an essential component for robotic manipulation and has been investigated for decades. Prior work on grasping often assumes that a sufficient amount of training data is available for learning and planning robotic grasps. However, constructing such an exhaustive training dataset is very challenging in practice, and it is desirable that a robotic system can autonomously learn and improves its grasping strategy. Although recent work has presented autonomous data collection through trial and error, such methods are often limited to a single grasp type, e.g. vertical pinch grasp. To address these issues, we present a hierarchical policy search approach for learning multiple grasping strategies. To leverage human knowledge, multiple grasping strategies are initialized with human demonstrations. In addition, a database of grasping motions and point clouds of objects is also autonomously built upon a set of grasps given by a user. The problem of selecting the grasp location and grasp policy is formulated as a bandit problem in our framework. We applied our reinforcement learning to grasping both rigid and deformable objects. The experimental results show that our framework autonomously learns and improves its performance through trial and error and can grasp previously unseen objects with a high accuracy.     

Tactile data modeling and interpretation for stable grasping and manipulation

This paper is devoted to present the latest results on the exploitation of the force/tactile sensor developed by the authors in terms of modeling and interpretation of the data provided by the device. An analytical nonlinear model of the elastically deformable sensor is derived and validated, which allows to reconstruct the position and orientation of the surface in contact with a rigid object on the basis of the sensor signals. The reconstruction is performed via an Extended Kalman Filter able to counteract the measurement noise and to handle the nonlinearity of the model at the same time. The contact plane position and orientation information together with the contact force vector measured by the sensor are used to estimate the physical parameter most relevant to manipulation control purposes: the friction coefficient. A slippage control algorithm is presented which exploits the estimated friction and a novel slipping detection algorithm is proposed to cope with the unavoidable uncertainties of the real world and its effectiveness is experimentally proved in comparison with the existing techniques.     

Development of multi-fingered dexterous hand for grasping manipulation

In this paper, a developed multi-fingered dexterous hand with flexible tactile skin is described. The dexterous hand has 5-fingers with 6-DOFs and each finger is equipped with a small harmonic drive gear and a fine high-power mini actuator. To achieve the goal of grasping with high accuracy, each fingertip is covered with the tactile array sensors for determination of the force between the finger and the grasped object. Some preliminary experiments are conducted to illustrate the performance of the grasping of the developed dexterous hand.     

Foreword to the special issue on autonomous grasping and manipulation

This paper summarizes an ongoing research program to advance the state of the art in robotics: the U.S. Defense Advanced Research Projects Agency Autonomous Robotic Manipulation (ARM) program. The program began in 2010 with three tracks, which was later extended to four. The software track is developing intelligent control of manipulators to perform autonomous tasks, using local perception sensors. The hand track has developed rugged, dexterous, multi-fingered hands with significantly reduced costs. The arm track is working to reduce the cost of robot arms. And the outreach track is showcasing robot technology to the general public. Technology developed under the program is iteratively evaluated through a series of hands-off tests. To date, the ARM developers have performed beyond expectations, yielding outstanding hardware designs and robust manipulation software. The results of this program are expected to strengthen the general robotics community and transition technology to U.S. military efforts.     

Learning object,grasping and manipulation activities using hierarchical HMMs

This article presents a probabilistic algorithm for representing and learning complex manipulation activities performed by humans in everyday life. The work builds on the multi-level Hierarchical Hidden Markov Model (HHMM) framework which allows decomposition of longer-term complex manipulation activities into layers of abstraction whereby the building blocks can be represented by simpler action modules called action primitives. This way, human task knowledge can be synthesised in a compact, effective representation suitable, for instance, to be subsequently transferred to a robot for imitation. The main contribution is the use of a robust framework capable of dealing with the uncertainty or incomplete data inherent to these activities, and the ability to represent behaviours at multiple levels of abstraction for enhanced task generalisation. Activity data from 3D video sequencing of human manipulation of different objects handled in everyday life is used for evaluation. A comparison with a mixed generative-discriminative hybrid model HHMM/SVM (support vector machine) is also presented to add rigour in highlighting the benefit of the proposed approach against comparable state of the art techniques.     

Demonstration-based learning and control for automatic grasping

We present a method for automatic grasp generation based on object shape primitives in a Programming by Demonstration framework. The system first recognizes the grasp performed by a demonstrator as well as the object it is applied on and then generates a suitable grasping strategy on the robot. We start by presenting how to model and learn grasps and map them to robot hands. We continue by performing dynamic simulation of the grasp execution with a focus on grasping objects whose pose is not perfectly known.     

Robotic motion planning and manipulation in an uncalibratedenvironment

The authors analyze planning and control problems in “robotic manipulation” in an uncalibrated environment consisting of a PUMA 560 robotic manipulator, a rotating turntable equipped with an encoder and a CCD camera based vision sensor fixed permanently on the ceiling. It is assumed that a part with a known shape but unknown orientation is placed on the turntable which is rotating with an unknown motion dynamics. Furthermore, the calibration parameters are a priori assumed to be unknown. The objective is to track the rotating part with an a priori specified relative orientation. The task considered is of importance in various problems concerning industrial automation, such as part-feeding and tool-changing     

Synergy-based affordance learning for robotic grasping

In this paper, we present an affordance learning system for robotic grasping. The system involves three important aspects: the affordance memory, synergy-based exploration, and a grasping control strategy using local sensor feedback. The affordance memory is modeled with a modified growing neural gas network that allows affordances to be learned quickly from a small dataset of human grasping and object features. After being trained offline, the affordance memory is used in the system to generate online motor commands for reaching and grasping control of the robot. When grasping new objects, the system can explore various grasp postures efficiently in the low dimensional synergy space because the synergies automatically avoid abnormal postures that are more likely to lead to failed grasps. Experimental results demonstrated that the affordance memory can generalize to grasp new objects and predict the effect of the grasp (i.e., the tactile patterns).     

Study on interaction-induced symptoms with respect to virtual grasping and manipulation

Owing to the popularity of various hand tracking interfaces, there have been numerous applications developed to provide intuitive hand interaction with the virtual world. As users start with great anticipation, they end up with dissatisfaction due to difficulties of manipulation or physical tiredness coming very short. Although the task itself is rather trivial in a real life situation, it requires much effort in the virtual environment. We address this awkwardness as ‘VR interaction-induced fatigue symptom’ and hypothesize its causes based on our observations. We argue that the source of the fatigue comes from the restricted sensory information of the VR interfaces, and that users try to accommodate the missing sensory feedback by excessive motion leading to wrong posture or bad timing. We demonstrate our hypothesis by conducting experiments of two types of virtual interaction scenarios: object transport and 3D selection. Furthermore, by analyzing the behaviors of users' action collected from our experiment, we derive essential factors to be considered in designing VR applications, and propose a conceptual interaction model for orchestrating virtual grasping.     

Optimising assembly learning in older adults through the manipulation of instruction

The present investigation assessed the putative benefits of reducing instructions for older adults' learning of an assembly task. Young and older adults had to build a product by assembling six components. Two groups practiced following instruction methods that differed in the degree of explicit information they conveyed about the correct assembly order. After practice, retention, consolidation of performance (tested immediately after practice and on a separate day, respectively) and stability of performance (tested by introducing a concurrent second task) were assessed. Younger adults showed similar performance levels for both instruction methods. Older adults, however, showed similar retention but clearly weaker consolidation and stability of performance following less encompassing instructions. Contrary to expectations, enhancing the involvement of explicit processes allowed older adults to gain a more permanent and stable performance improvements. The findings are discussed relative to the characteristics of the assembly task.