Networked telemicromanipulation systems "Haptic Loupe"

In this paper, "Haptic Loupe" telemicromanipulation systems are proposed. We have developed telemicromanipulation systems that enable human operators to perform micro tasks, such as assembly or manufacturing without stress . These systems are based on a scaled bilateral teleoperation system between different structures. The systems are composed of an original six-degrees-of-freedom (6-DOF) parallel link manipulator to carry out micromanipulation and a 6-DOF haptic interface with force feedback. A parallel mechanism is adopted as a slave micromanipulator because of its good features of accuracy and stiffness. The haptic master interface is developed for micromanipulation systems. Haptic device system modeling and a model reference adaptive controller are implemented to compensate for friction forces, which spoil the free motion performance and force response isotropy of the system. Total system performance as a telemicromanipulator system is evaluated by performing some primitive manipulation tasks in a teleoperation experiment. Experimental results are presented and discussed.     

Performance evaluation of a tactile and kinesthetic finger feedback system for teleoperation

Teleoperation during a catastrophic event requires an interface that can perform under frequently changing circumstances caused by unpredictable and dangerous conditions. Thus, teleoperation interfaces are under active development to provide both visual and haptic feedback to the fingers. However, studies of teleoperation systems with finger haptic feedback based on force profiles are difficult to conduct because of interface limitations. Therefore, in this paper, we introduce an intuitive teleoperation interface, an anthropomorphic teleoperated robot, and a hand-wearable force-feedback system that provides various feedbacks to the fingers. We combined these systems to compare and evaluated the performance of tactile and kinesthetic finger feedback using two experiments: maintaining appropriate grip force for variably fragile objects and following a force trajectory that changed in real time. Ten subjects participated in the experiments. The results were analyzed using repeated measures analysis of variance. Feedback factors differed significantly. Provision of force feedback to the user’s finger was most effective in both teleoperation experiments.     

Supermedia-enhanced Internet-based telerobotics

This paper introduces new planning and control methods for supermedia-enhanced real-time telerobotic operations via the Internet. Supermedia is the collection of video, audio, haptic information, temperature, and other sensory feedback. However, when the communication medium used, such as the Internet, introduces random communication time delay, several challenges and difficulties arise. Most importantly, random communication delay causes instability, loss of transparency, and desynchronization in real-time closed-loop telerobotic systems. Due to the complexity and diversity of such systems, the first challenge is to develop a general and efficient modeling and analysis tool. This paper proposes the use of Petri net modeling to capture the concurrency and complexity of Internet-based teleoperation. Combined with the event-based planning and control method, it also provides an efficient analysis and design tool to study the stability, transparency, and synchronization of such systems. In addition, the concepts of event transparency and event synchronization are introduced and analyzed. This modeling and control method has been applied to the design of several supermedia-enhanced Internet-based telerobotic systems, including the bilateral control of mobile robots and mobile manipulators. These systems have been experimentally implemented in three sites test bed consisting of robotic laboratories in the USA, Hong Kong, and Japan. The experimental results have verified the theoretical development and further demonstrated the stability, event transparency, and event synchronization of the systems.     

Development of micromanipulator and haptic interface for networkedmicromanipulation

Telemicromanipulation systems with haptic feedback, which are connected through a network, are proposed. It is based on scaled bilateral teleoperation systems between different structures. These systems are composed of an original 6 degree of freedom (DOF) parallel link manipulator to carry out micromanipulation and a 6-DOF haptic interface with force feedback. A parallel mechanism is adopted as a slave micromanipulator because of its good features of accuracy and stiffness. The system modeling and control of the parallel manipulator system are conducted. Parallel manipulator feasibility as a micromanipulator, positioning accuracy and device control characteristics are investigated. A haptic master interface is developed for micromanipulation systems. System modeling and a model reference adaptive controller are applied to compensate friction force, which spoils free motion performance and force response isotropy of the haptic interface. These systems aim to make the micromanipulation more productive constructing a better human interface through the microenvironment force and scale expansion     

A survey of environment-, operator-, and task-adapted controllers for teleoperation systems

Bilateral haptic teleoperation systems allow humans to perform complex tasks in a remote or inaccessible environment, while providing haptic feedback to the human operator. The incorporation of online gained environment-, operator-, or task-specific (EOT) information in the controller structure can lead to significant improvements in robustness, task performance, feeling of presence, or fidelity without compromising stability. This article provides a classification as well as a survey of approaches, called EOT-adapted controllers, which have been developed in this area. A discussion of improvements and requirements is provided for each method. The performed analysis indicates that several methods require the usage of additional sensors or are based on accurate model assumptions. The benefit of EOT-adapted controllers is mostly application-dependent, as each method focuses on the improvement of a specific aspect like coping with time delay or avoiding forbidden regions.     

Enabling 4-DoF hand guidance using a portable haptic device exerting tangential force on the user's finger pads

Body-grounded kinesthetic haptic devices can provide cues for movement in multiple degrees of freedom by exerting forces directly on users, as in dexterous robot teleoperation tasks. However, these haptic devices have limited workspaces, can destabilize a teleoperation control loop, and can be expensive. Portable haptic devices can approximate the sensations of a kinesthetic device by exploiting diverse human sense of touch principles without these shortcomings. Our goal is to analyze the feasibility of hand guidance (HG) using tangential force stimuli. Here we reveal and quantify users’ interpretation of simultaneous tactile stimulation (STS) applied to multiple finger pads of the same hand. We completed an extensive experiment on different users to reveal a maximum number of understandable cues which can be used as movement commands for HG. As expected, many tactile stimuli tested were meaningless for users, but a few could be clearly interpreted — we call these “intuitive movement cues”. For the experiment, we designed a device that can be held in the palm and exerts tactile stimuli to the user's finger pads on the thumb and index fingers, or the thumb and middle fingers. We performed two studies in which we identified the extent of salience of different movement cues. In particular, commands to redirect the hand position and orientation in four axes: moving forward/backward, wrist twisting right/left (rotate clockwise/counter-clockwise), moving right/left, and wrist tilting up/down (rotate upwards/downwards). The results revealed that this approach provided 7 intuitive directional movement cues for relative HG in 3D space. The proposed HG principle is promising for applications such as robotic surgery training, laparoscopic training, and needle insertion training, during which surgical trainees must learn dexterous hand movements involving motion paths. There are many applications for 3D movement guidance outside the medical domain that could benefit from this haptics technology, including training for precise manipulation and assembly tasks, augmented teleoperation, and communication during shared control in collaborative human-machine systems.     

Force display using a hybrid haptic device composed of motors and brakes

In the virtual environment, force feedback to the human operator makes virtual experiences more realistic. However, the force feedback using active actuators such as motors can make the system active and sometimes unstable. To ensure the safe operation and enhance the haptic feeling, system stability should be guaranteed. Both active actuators such as motors and passive ones such as brakes are commonly used for haptic devices. Motors can generate a torque in any direction, but they can make the system active and thus, sometimes unstable during operation. On the other hand, brakes can generate a torque only against their rotation, but they dissipate energy during operation and this dissipation makes the system intrinsically stable. Consequently, motors and brakes are complementary to each other. In this research, a two degree-of-freedom (DOF) haptic device equipped with motors and brakes is designed, in which each DOF is actuated by a pair of motor and brake. Simultaneous operation of motors and brakes is analyzed. Models for some environments, virtual wall contact and frictional effect, are proposed. The results for the hybrid haptic system are compared with those for the active haptic system and the passivity based control system. The experimental results show that the hybrid haptic device is more suited to some applications than the other haptic systems.     

Air Permeable Vibrotactile Actuators for Wearable Wireless Haptics

Vibrotactile actuators can evoke mechanical stimulations on human skins to induce haptic feedbacks for various human machine interaction applications. However, efforts toward their practical usages encounter several engineering challenges, including wearable comfortability and output abilities. Here, air permeable actuators are developed and embedded in common fabrics for vibrotactile actuation, achieving excellent air permeability of 108 L m⁻² s⁻¹, low preload requirement of 10 mN, high output sensitivity of 0.2 mN/V, and good mechanical durability by surviving 11 million testing cycles. As demonstration examples, a wireless haptic feedback glove is shown to distinguish 32 different English characters and symbols with an overall accuracy of 97.8%, and large size actuators (10 × 10 cm²) are also proved for providing haptic feedback for parts of human body. As such, the proposed system opens a new class of wearable vibrotactile actuators for potential applications in wide fields of metaverse, teleoperation, smart textiles, and robotics.     

Force controlled and teleoperated endoscopic grasper for minimallyinvasive surgery-experimental performance evaluation

Minimally invasive surgery generates new user interfaces which create visual and haptic distortion when compared to traditional surgery. In order to regain the tactile and kinesthetic information that is lost, a computerized force feedback endoscopic surgical grasper (FREG) was developed with computer control and a haptic user interface. The system uses standard unmodified grasper shafts and tips. The FREG can control grasping forces either by surgeon teleoperation control, or under software control. The FREG performance was evaluated using an automated palpation function (programmed series of compressions) in which the grasper measures mechanical properties of the grasped materials. The material parameters obtained from measurements showed the ability of the FREG to discriminate between different types of normal soft tissues (small bowel, lung, spleen, liver, colon, and stomach) and different kinds of artificial soft tissue replication materials (latex/silicone) for simulation purposes. In addition, subjective tests of ranking stiffness of silicone materials using the FREG teleoperation mode showed significant improvement in the performance compared to the standard endoscopic grasper. Moreover, the FREG performance was closer to the performance of the human hand than the standard endoscopic grasper. The FREG as a tool incorporating the force feedback teleoperation technology may provide the basis for application in telesurgery, clinical endoscopic surgery, surgical training, and research.     

Stable kinematic teleoperation of wheeled mobile robots with slippage using time-domain passivity control

Wheel slippage creates control challenges for wheeled mobile robots (WMR). This paper proposes a new method for haptic teleoperation control of a WMR with longitudinal slippage by using the time-domain passivity control (TDPC) approach. We show the potential non-passivity for the environment termination caused by the slippage dynamics. The utilized TDPC approach maintains the passivity of teleoperation system terminations through a passivity observer and a passivity controller at the environment termination. The teleoperation controllers are then simply constrained by Llewellyn's absolute stability criterion for closed-loop stability purposes. Experiments with the proposed controller demonstrate that it can result in stable bilateral teleoperation with a satisfactory tracking performance with TDPC.     

Bilateral robot system on the real-time network structure

This paper presents a bilateral robot system, which is driven by the static friction-free drive system and implemented on the real-time network structure. The goal is to realize a force reflecting bilateral teleoperation with haptic impression transmission over computer networks. The paper considers two subjects relating to the bilateral robot. The first is static friction, which degrades the performance of manipulation and results in a poor haptic impression. A new transmission mechanism named twin drive system developed by the authors resolves this problem. The transmission mechanism, which resembles the differential gear of automobiles, is essentially free of static friction. This static-friction-free motion greatly contributes to the broad range of motion control applications. The second subject is the time delay of the network, which may cause serious problems such as instability of the feedback system. To avoid such delay, the authors developed a new real-time network protocol stack (RTNP). The detailed mechanism of the twin drive system and architecture of the RTNP are presented, and the control scheme and experimental results are also shown.     

Teleoperated touch feedback from the surfaces at the nanoscale: modeling and experiments

In this paper, a teleoperated nanoscale touching system is proposed, and continuum nanoscale contact mechanics models are introduced. The tele-nanorobotic system consists of a piezoresistive nanoprobe with a sharp tip as the nanorobot and force-topology sensor, a custom-made 1-degree-of-freedom haptic device for force-feedback, three-dimensional (3D) virtual reality (VR) graphics display of the nano world for visual feedback, and a force-reflecting servo type scaled teleoperation controller. Using this system, one-dimensional and 3D touching experiments and VR simulations are realized. Scaling of nano-forces is one of the major issues of the scaled teleoperation system since nanometer scale forces are dominated by surface forces instead of inertial forces as in the macro world. As the force scaling approach, a heuristic rule is introduced where nano-forces are linearly scaled with an experimentally determined scaling parameter. Simulation results and preliminary experiments of touching silicon and InAs quantum dot nanostructures show that adhesion forces at the nanoscale can be felt repeatedly at the operator's hand, and the proposed system enables the nanoscale surface topography and contact/noncontact nano-force feedback.     

On Extending the Wave Variable Method to Multiple-DOF Teleoperation Systems

It is well known that providing a human operator with contact force information can significantly improve task performance in a teleoperation system. Unfortunately, time delay is a serious problem for such systems. Even a small time delay in a bilateral teleoperation system will generally degrade the system's performance and cause instability. Consequently, without some form of compensation for time delay, latencies in a teleoperation system would preclude the use of force feedback. Fortunately, there are approaches based on scattering theory and passivity that can compensate for time delay and allow the use of force feedback in teleoperation systems with latencies. In particular, the wave variable method is a passivity-based approach that guarantees stability for any fixed time delay. Since its introduction, the wave variable method has been augmented with predictors to compensate for variable time delay. The wave variable formalism has also been extended to multiple-DOF systems by replacing scalar damping constants with a family of impedance matrices. In this paper, the authors generalize this last approach to include a larger family of impedance matrices. The paper includes a complete derivation of the extended family of impedance matrices as well as simulation and experimental results to illustrate the approach.      

Teleoperation of a robot manipulator using a vision-based human-robot interface

Remote teleoperation of a robot manipulator by a human operator is often necessary in unstructured dynamic environments when human presence at the robot site is undesirable. Mechanical and other contacting interfaces used in teleoperation require unnatural human motions for object manipulation tasks or they may hinder human motion. Previous vision-based approaches have used only a few degrees of freedom for hand motion and have required hand motions that are unnatural for object manipulation tasks. This paper presents a noncontacting vision-based method of robot teleoperation that allows a human operator to communicate simultaneous six-degree-of-freedom motion tasks to a robot manipulator by having the operator perform the three-dimensional human hand-arm motion that would naturally be used to complete an object manipulation task. A vision-based human-robot interface is used for communication of human motion to the robot and for feedback of the robot motion and environment to the human operator. Teleoperation under operator position control was performed with high accuracy in object placement on a target. Semi-autonomous traded and shared control using robot-vision guidance aided in achieving a more accurate positioning and orientation of the end-effector for object gripping tasks.     

Adaptive impedance control of a haptic interface

Teleoperation enables an operator to manipulate remote objects. One of the main goals in teleoperation research is to provide the operator with the feeling of the telepresent object and of being present at the remote site. In order for this to happen, a master robot must be designed as a bilateral control system that can transmit position commands to a slave robot and reflect the interaction force. A newly proposed adaptive impedance algorithm is applied to the force control of a haptic interface that has been developed as a master robot. With the movement of the haptic interface for position command generation, the impedance between an operator and the haptic interface varies dynamically. When the impedance parameters and the dynamics of the haptic interface are known precisely, many model based control theories and methods can be used to control the interface accurately. However, due to the parameters’ variations and the uncertainty in the dynamic model, it is difficult to control the interface precisely. Therefore, this paper proposes a new adaptive impedance control algorithm and experimentally verifies the effectiveness of the algorithm for control of the haptic interface.     

Wearable Soft Technologies for Haptic Sensing and Feedback

Virtual reality (VR) and augmented reality (AR) systems have garnered recent widespread attention due to increased accessibility, functionality, and affordability. These systems sense user inputs and typically provide haptic, audio, and visual feedback to blend interactive virtual environments with the real world for an enhanced or simulated reality experience. With applications ranging from immersive entertainment, to teleoperation, to physical therapy, further development of this technology has the potential for impact across multiple disciplines. However, VR/AR devices still face critical challenges that hinder integration into everyday life and additional applications; namely, the rigid and cumbersome form factor of current technology that is incompatible with the dynamic movements and pliable limbs of the human body. Recent advancements in the field of soft materials are uniquely suited to provide solutions to this challenge. Devices fabricated from flexible and elastic bio-compatible materials have significantly greater compatibility with the human body and could lead to a more natural VR/AR experience. This review reports state-of-the-art experimental studies in soft materials for wearable sensing and haptic feedback in VR/AR applications, explores emerging soft technologies for on-body devices, and identifies current challenges and future opportunities toward seamless integration of the virtual and physical world.     

iBotGuard: an Internet-based Intelligent Robot security system using Invariant Face Recognition against intruder

One crucial application of intelligent robotic systems is remote surveillance using a security robot. A fundamental need in security is the ability to automatically verify an intruder into a secure or restricted area, to alert remote security personnel, and then to enable them to track the intruder. In this article, we propose an Internet-based security robot system. The face recognition approach possesses "invariant" recognition characteristics, including face recognition where facial expressions, viewing perspectives, three-dimensional poses, individual appearance, and lighting vary and occluding structures are present. The experiment uses a 33.6-kb/s modem Internet connection to successfully remotely control a mobile robot, proving that the streaming technology-based approach greatly improves the "sensibility" of robot teleoperation. This improvement ensures that security personnel can effectively and at low cost use the Internet to remotely control a mobile robot to track and identify a potential intruder.     

Haptic joystick with hybrid actuator using air muscles and spherical MR-brake

In this research, a new 2-DOF hybrid actuator concept is explored as a powerful and compact alternative to conventional haptic actuators. The actuator combines a spherical MR-brake and three air muscles and is integrated into a joystick that can apply forces in two degrees-of-freedom. The air muscles are used to create high active forces in a compact volume and the brake compensates for the “spongy” feeling associated with air muscles. To decrease the overall size of the system an inertial measurement unit has been implemented as a position measurement solution. As high as 16 N of total force output could be achieved at the tip of the joystick. Also, up to 16 times improvement in the stable virtual wall stiffness was obtained when the MR-brake was used to compensate for force errors. Experiments with an impedance-based haptic controller with force-feedback gave satisfactory wall following performance. This device can be employed in applications including computer games, military or medical training applications, rehabilitation and in teleoperation of equipment where high force feedback in 2-DOF in a compact work volume may be desirable while interacting with rigid or elastic virtual objects.     

High-frequency acceleration feedback in wave variable telerobotics

The human hand is very sensitive to the high-frequency accelerations produced by tool contact with a hard object, yet most time delayed telerobots neglect this feedback band entirely in order to achieve stability. We present a control architecture that both incorporates this important information and provides the ability to scale and shape it independently of the low-frequency force feedback. Leveraging the clean power flows afforded by wave variables, this augmented controller preserves the passivity of any environment that it renders to the user, but is not subject to the limitations of being passive itself. This architecture guarantees stability in the presence of communication delay while achieving a level of feedback not possible with a passive controller. We show experimentally that this feedback augmentation and shaping can present a high-frequency acceleration profile to the user's hand that is similar to that experienced by the slave end effector. Two simple user studies also show that the feedback augmentation improves the user's perception, performance, and confidence with the given tasks. We anticipate that these natural haptic cues will make teleoperative systems easier to use and thus more widely applicable.     

A control-system architecture for robots used to simulate dynamicforce and moment interaction between humans and virtual objects

Haptic or kinesthetic feedback is essential in many important virtual reality and telepresence applications. Previous research focuses on simulating static forces such as those encountered when interacting with a stiff object such as a wall. Past studies usually employ custom-made devices that are not readily available to other researchers. Consequently, many of the results found in the haptic feedback literature cannot be replicated independently. With experimental results, the paper demonstrates that “off the shelf,” general purpose robotics equipment can be incorporated into an effective haptic/kinesthetic feedback system. Such a system can accommodate a wide variety of virtual reality applications including training and telerobotics. An admittance control scheme is utilized, which enables the simulation of dynamic force and moment interaction as well as contact with stiff objects. The paper shows that the mechanical deficiencies (e.g., friction, inertia, and backlash) often associated with general purpose manipulators can be overcome with a suitable control system architecture