A computer vision assisted system for autonomous forklift vehicles in real factory environment

Industry 4.0 is an important trend in factory automation nowadays. Among the Automated-Storage-and-Retrieval-System (ASRS) is one of the most important issues for industry. It is widely used in a variety of industries for a variety of storage applications in factories and warehouses. However, the cost of constructing an ASRS is so high that most small/medium enterprises cannot afford it. A forklift system is a cheaper alternative to a complicated ASRS. In this work, a new pallet detection method that uses an Adaptive Structure Feature (ASF) and Direction Weighted Overlapping (DWO) ratio to allow forklifts to pick up a pallet is proposed, using a monocular vision system on the forklift. Combining the ASF and DWO ratio for pallet detection, the proposed method removes most of the non-stationary (dynamic) background and significantly increases the processing efficiency. A Haar like-based Adaboost scheme uses an AS for pallets algorithm to detect pallets. It detects the pallet in a dark environment. Finally, by calculating the DWO ratio between the detected pallets and tracking records, it avoids erroneous candidates during object tracking. Therefore, this work improves the pallet detection to solve the problem with an effective design. As results show that the hybrid algorithms that are proposed in this work increase the average pallet detection rate by 95 %.     

Arrow R-CNN for handwritten diagram recognition

We address the problem of offline handwritten diagram recognition. Recently, it has been shown that diagram symbols can be directly recognized with deep learning object detectors. However, object detectors are not able to recognize the diagram structure. We propose Arrow R-CNN, the first deep learning system for joint symbol and structure recognition in handwritten diagrams. Arrow R-CNN extends the Faster R-CNN object detector with an arrow head and tail keypoint predictor and a diagram-aware postprocessing method. We propose a network architecture and data augmentation methods targeted at small diagram datasets. Our diagram-aware postprocessing method addresses the insufficiencies of standard Faster R-CNN postprocessing. It reconstructs a diagram from a set of symbol detections and arrow keypoints. Arrow R-CNN improves state-of-the-art substantially: on a scanned flowchart dataset, we increase the rate of recognized diagrams from 37.7 to 78.6%.

     

快速多域卷积神经网络和光流法融合的目标跟踪

Aiming at the problem of slow speed of the convolutional neural network target tracking algorithm, a target tracking algorithm combining fast multi-domain convolutional neural network (Faster MDNet) and optical flow method is proposed. The optical flow method is used to obtain the moving state of the target, and the preliminary selection box is used as the tracking target position. Then, the preliminary selection box is used as the input of Faster MDNet, and Faster MDNet is used as the detector to obtain the exact position and bounding box of the tracking target. Experiments on the target tracking benchmark data set VOT2014 prove that the algorithm’s online tracking speed is increased by 8 times and the accuracy is improved by 10%.     

考虑混合型号托盘的托盘共用调度随机规划模型

如何将托盘科学地在系统内调度是目前托盘共用系统管理者们亟待解决的问题。面对各类参数的随机性和多种多样的托盘,管理者很难仅凭经验做出科学的决策。利用随机机会约束规划的方法,构建了考虑混合型号托盘的托盘共用系统调度随机规划模型,使用确定性等价转化的方法将机会约束转化为了其确定等价形式,通过算例进行了数值求解和分析,验证了模型的有效性,提出了决策策略建议。     

Electromagnetic modular Smart Surface architecture and control in a microfactory context

This paper presents an electromagnetic conveyance system called electromagnetic modular Smart Surface (emSS) permitting to move pallets on a planar surface in a microfactory context. The proposed surface concept allows flexibility in reconfiguring the system layout along with product routing. The possibilities of accurate positioning of the moving pallet and controlling multiple pallets on the surface make the emSS suitable for reconfigurable and flexible manufacturing systems. However, the emSS control needs to be robust and scalable to adapt the changes in manufacturing systems. A framework is therefore defined to monitor and control the emSS by simulation or in-line. It allows to define product routing on the emSS by satisfying numerous requirements such as reduction in energy consumption, collision avoidance, etc., and to minimize the human interventions by changing product routing when emSS component failures occur. A first experiment realized on an emSS prototype, allowed to compare two paths strategies regarding cost function linked to energy consumption and velocities. Two other studies exploit the emSS modeling in terms of pallet path generation and simulation of collision avoidance.     

Learning control of mobile robots using a multiprocessor system

A real-time multiprocessor system is proposed for the solution of the tracking problem of mobile robots operating in a real context with environmental disturbances and parameter uncertainties. The proposed control scheme utilizes multiple models of the robot for its identification in an adaptive and learning control framework. Radial Basis Function Networks (RBFNs) are considered for the multiple models in order to exploit the net non-linear approximation capabilities for modeling the kinematic behavior of the vehicle and for reducing unmodeled contributions to tracking errors. The training of the nets and the tests of the achieved control performance have been done in a real experimental setup. The proposed control architecture improves the robot tracking performance achieving fast and accurate control actions in presence of large and time-varying uncertainties in dynamical environments. The experimental results are satisfactory in terms of tracking errors and computational efforts.     

移动机器人运动目标跟踪系统设计

利用SONYEV-D31摄像机和自主研发的摄像机控制模块,构建了一套主动视觉子系统,并将该子系统应用于RIRA-Ⅱ型移动机器人上,实现了移动机器人运动目标自动跟踪功能。RIRA-Ⅱ移动机器人采用了由一组分布式行为模块和集中命令仲裁器组成的基于行为的分布式控制体系结构。各行为模块基于领域知识通过反应方式产生投票,由仲裁器产生动作指令,机器人完成相应的动作。在设置了障碍、窄通道以及模拟墙体的复杂环境下进行运动目标跟踪实验,实验表明运动目标跟踪系统运行可靠,具有较高的鲁棒性。     

Fast RGB-D people tracking for service robots

Service robots have to robustly follow and interact with humans. In this paper, we propose a very fast multi-people tracking algorithm designed to be applied on mobile service robots. Our approach exploits RGB-D data and can run in real-time at very high frame rate on a standard laptop without the need for a GPU implementation. It also features a novel depth-based sub-clustering method which allows to detect people within groups or even standing near walls. Moreover, for limiting drifts and track ID switches, an online learning appearance classifier is proposed featuring a three-term joint likelihood. We compared the performances of our system with a number of state-of-the-art tracking algorithms on two public datasets acquired with three static Kinects and a moving stereo pair, respectively. In order to validate the 3D accuracy of our system, we created a new dataset in which RGB-D data are acquired by a moving robot. We made publicly available this dataset which is not only annotated by hand, but the ground-truth position of people and robot are acquired with a motion capture system in order to evaluate tracking accuracy and precision in 3D coordinates. Results of experiments on these datasets are presented, showing that, even without the need for a GPU, our approach achieves state-of-the-art accuracy and superior speed.     

Real-time seam tracking for robotic laser welding using trajectory-based control

In this paper a real-time seam tracking algorithm is proposed that can cope with the accuracy demands of robotic laser welding. A trajectory-based control architecture is presented, which had to be developed for this seam tracking algorithm. Cartesian locations (position and orientation) are added to the robot trajectory during the robot motion. In this way, sensor information obtained during the robot motion is used to generate the robot trajectory while moving. Experiments have been performed to prove the tracking capabilities of the seam tracking algorithm.     

On-road vehicle detection in varying weather conditions using faster R-CNN with several region proposal networks

Developing automated systems to detect and track on-road vehicles is a demanding research area in Intelligent Transportation System (ITS). This article proposes a method for on-road vehicle detection and tracking in varying weather conditions using several region proposal networks (RPNs) of Faster R-CNN. The use of several RPNs in Faster R-CNN is still unexplored in this area of research. The conventional Faster R-CNN produces regions-of-interest (ROIs) through a single fixed sized RPN and therefore cannot detect varying sized vehicles, whereas the present investigation proposes an end-to-end method of on-road vehicle detection where ROIs are generated using several varying sized RPNs and therefore it is able to detect varying sized vehicles. The novelty of the proposed method lies in proposing several varying sized RPNs in conventional Faster R-CNN. The vehicles have been detected in varying weather conditions. Three different public datasets, namely DAWN, CDNet 2014, and LISA datasets have been used to evaluate the performance of the proposed system and it has provided 89.48%, 91.20%, and 95.16% average precision on DAWN, CDNet 2014, and LISA datasets respectively. The proposed system outperforms the existing methods in this regard.

     

分块多特征目标描述子的移动机器人目标跟踪

为解决机器人目标跟踪过程中的遮挡和外观改变等问题, 提出一种分块多特征描述子的方法. 该方法将候选样本分块, 提取图像片的深度、颜色、纹理特征来表示目标构造检测器. 结合目标与机器人的运动构造运动卡尔曼滤波器(MEKF) 作为跟踪器. 跟踪过程中根据目标深度信息调整其尺寸, 结合深度特征及图像片外观相似度进行检测并处理遮挡. 实验结果表明, 该算法对目标的尺度变化、光照改变和遮挡现象具有较强的鲁棒性.

     

一种平板车装载问题的启发式算法

提出平板车装载问题的一各上启发式方法,不仅能够从几种型号的平板车中选择出一种装载性能较好的平板车,而且能够将具有不同特性的不同尺寸的物品放在该型号平板车中,它能够在满足一些约束条件下求得一种可行的装载方案,实例充分证明了该算法的有效性和实用性,能够直接用于实际生活中。     

A neural network-based real-time robot tracking controller using position sensitive detectors

Abstract: A real-time visual servo tracking system for an industrial robot has been developed. Instead of a charge coupled device (CCD), a position sensitive detector (PSD) is used as the real-time vision sensor due to its fast response (the light position is transduced to analogue current). A neural network learns the complex association between the 3D object position and its sensor reading, and uses it to track that object, either moving or stationary. It also turns out that this scheme lends itself to a user-friendly way to teach workpaths for industrial robots. Furthermore, for real-time use of the neural net, an efficient neural network architecture has been developed based on the concept of input space partitioning and local learning. Real experiments indicate the system's characteristics of fast processing and learning as well as optimal usage of network resources.     

Real-Time Dynamic Visual Tracking Using PSD Sensors and Extended Trapezoidal Motion Planning

A real-time visual servo tracking system for an industrial robot has been implemented using PSD (Position Sensitive Detector) cameras, neural networks, and an extended trapezoidal motion planning method. PSD and directly transduces the light's projected position on its sensor plane into an analog current and lends itself to fast real-time tracking. A neural network, after proper training, transforms the PSD sensor reading into a 3D position of the target, which is then input to an extended trapezoidal motion planning algorithm. This algorithm implements a continuous motion update strategy in response to an ever-changing sensor information from the moving target, while greatly reducing the tracking delay. This planning method is found to be very useful for sensor-based control such as moving target tracking or weld-seam tracking in which the robot needs to change its motion in real time in response to incoming sensor information. Further, for real-time usage of the neural net, a new architecture called LANN (Locally Activated Neural Network) has been developed based on the concept of CMAC input partitioning and local learning. Experimental evidence shows that an industrial robot can smoothly track a moving target of unknown motion with speeds of up to 1 m/s and with oscillation frequency up to 5 Hz.     

基于“非伪”理论的非完整移动机器人轨迹跟踪控制

"非伪"控制是一种基于数据驱动的无模型控制方法,它根据输入-输出数据进行在线学习,计算与当前系统状态相匹配的控制量并作用于系统,以获得系统所要求的动静态品质,并以此检验系统是否满足该性能指标.基于"非伪"控制理论,研究了移动机器人的轨迹跟踪控制问题.根据非完整移动机器人的动态方程,采用"非伪"控制,直接作用于移动机器人的控制输入,使移动机器人能快速、准确地跟踪期望轨迹.     

Real-time system for elliptical-based face detection

In this paper, we present a real-time ellipse detector in gray scale images with a new multiple stage architecture based on a 3-accumulator version of the Fast Hough Transform with a previous Canny edge extraction. The system can be applied to detect different elliptical objects and is robust to incomplete ellipses, cluttered backgrounds and illumination changes. It achieves 12 frames per second on a PC Pentium 4, 2.80 GHz. Experimental results, focusing on faces, with both static and video images are showed. The presented ellipse detector can be used as a preprocessing module in a face tracking or recognition application.     

Using dialog and human observations to dictate tasks to a learning robot assistant

Robot assistants need to interact with people in a natural way in order to be accepted into people’s day-to-day lives. We have been researching robot assistants with capabilities that include visually tracking humans in the environment, identifying the context in which humans carry out their activities, understanding spoken language (with a fixed vocabulary), participating in spoken dialogs to resolve ambiguities, and learning task procedures. In this paper, we describe a robot task learning algorithm in which the human explicitly and interactively instructs a series of steps to the robot through spoken language. The training algorithm fuses the robot’s perception of the human with the understood speech data, maps the spoken language to robotic actions, and follows the human to gather the action applicability state information. The robot represents the acquired task as a conditional procedure and engages the human in a spoken-language dialog to fill in information that the human may have omitted.     

Vision-based short range interaction between a personal service robot and a user

Interaction between a personal service robot and a human user is contingent on being aware of the posture and facial expression of users in the home environment. In this work, we propose algorithms to robustly and efficiently track the head, facial gestures, and the upper body movements of a user. The face processing module consists of 3D head pose estimation, modeling nonrigid facial deformations, and expression recognition. Thus, it can detect and track the face, and classify expressions under various poses, which is the key for human–robot interaction. For body pose tracking, we develop an efficient algorithm based on bottom-up techniques to search in a tree-structured 2D articulated body model, and identify multiple pose candidates to represent the state of current body configuration. We validate these face and body modules in varying experiments with different datasets, and the experimental results are reported. The implementation of both modules can run in real-time, which meets the requirement for real-world human–robot interaction task. These two modules have been ported onto a real robot platform by the Electronics and Telecommunications Research Institute.     

Task-space synchronisation of nonlinear teleoperation with time-varying delays and actuator saturation

ABSTRACT

In a nonlinear teleoperation system controlled for task-space position tracking, while the time-varying delay in the communication channel has been addressed, the actuator saturation has not been taken into account yet. Considering that in practice, the actuator saturation is a serious constraint, disregarding it in the controller design stage can cause problems. In this paper, we have proposed a control framework to ensure end-effectors position tracking while satisfying sub-task control in the presence of the nonlinear dynamics for the telemanipulators, bounded time-varying delays in the communication channels and saturation in the actuators. We have shown that in free motion and when the operator applies a bounded force to the local robot, the proposed controller not only guarantees the position convergence of the end-effectors but also guarantees the accomplishment of the sub-task control. The efficiency of the proposed control algorithm is validated showing a number of numerical simulations.     

The effects of operator position,pallet orientation,and palletizing condition on low back loads in manual bag palletizing operations

Many mining commodities are packaged and shipped using bags. Small bags are typically loaded onto pallets for transport and require a significant amount of manual handling by workers. This specific task of manual bag handling has been associated with the development of musculoskeletal disorders (MSDs), especially low back disorders. This study evaluates the biomechanical demands of different work layouts when performing manual palletizing of small bags, and evaluates the biomechanical stresses associated with different stacking techniques. Results indicate that peak forward bending moments as well as spinal compression and shear forces are higher when the pallet is situated at the side of the conveyor as opposed to the end of the conveyor. At low levels of the pallet, controlled bag placement results in higher peak forward bending moments than stacking at higher levels and when dropping the bag to lower levels. The results of this study will be used to inform the development of an audit tool for bagging operations in the mining industry.Relevance to industryIn many cases for workers loading small bags, compression forces exceed the NIOSH criterion of 3400 N. Orientation of the pallet has a significant impact on spinal compression, and positioning the pallet at the end of the conveyor reduces the estimated compressive loading on the lumbar spine by approximately 800 N.