Unmanned aerial vehicles UAVs attitude,height, motion estimation and control using visual systems

This paper presents an implementation of an aircraft pose and motion estimator using visual systems as the principal sensor for controlling an Unmanned Aerial Vehicle (UAV) or as a redundant system for an Inertial Measure Unit (IMU) and gyros sensors. First, we explore the applications of the unified theory for central catadioptric cameras for attitude and heading estimation, explaining how the skyline is projected on the catadioptric image and how it is segmented and used to calculate the UAV’s attitude. Then we use appearance images to obtain a visual compass, and we calculate the relative rotation and heading of the aerial vehicle. Additionally, we show the use of a stereo system to calculate the aircraft height and to measure the UAV’s motion. Finally, we present a visual tracking system based on Fuzzy controllers working in both a UAV and a camera pan and tilt platform. Every part is tested using the UAV COLIBRI platform to validate the different approaches, which include comparison of the estimated data with the inertial values measured onboard the helicopter platform and the validation of the tracking schemes on real flights.     

EKF Based Attitude Estimation and Stabilization of a Quadrotor UAV Using Vanishing Points in Catadioptric Images

In recent years, Unmanned Air Vehicles (UAVs) have become more and more important. These vehicles are employed in many applications from military operations to civilian tasks. Under situations where global positioning system (GPS) and inertial navigation system (INS) do not function, or as an additional sensor, computer vision can be used. Having 360° view, catadioptric cameras might be very useful as they can be used as measurement units, obstacle avoidance sensors or navigation planners. Although many innovative research has been done about this camera, employment of such cameras in UAVs is very new. In this paper, we present the use of catadioptric systems in UAVs to estimate vehicle attitude using parallel lines that exist on many structures in an urban environment. After explanation of the algorithm, the UAV modeling and control will be presented. In order to increase the estimation and control speed an Extended Kalman Filter (EKF) and multi-threading are used and speeds up to 40 fps are obtained. Various simulations have been done to present the effectiveness of the estimation algorithms as well as the UAV controllers. A custom test stand has been designed to perform successful experiments on the UAV. Finally, we will present the experiments and the results of the estimation and control algorithms on a real model helicopter. EKF based attitude estimation and stabilization using catadioptric images has found to be a reliable alternative to other sensor usage.     

Vision Based UAV Attitude Estimation: Progress and Insights

Unmanned aerial vehicles (UAVs) are increasingly replacing manned systems in situations that are dangerous, remote, or difficult for manned aircraft to access. Its control tasks are empowered by computer vision technology. Visual sensors are robustly used for stabilization as primary or at least secondary sensors. Hence, UAV stabilization by attitude estimation from visual sensors is a very active research area. Vision based techniques are proving their effectiveness and robustness in handling this problem. In this work a comprehensive review of UAV vision based attitude estimation approaches is covered, starting from horizon based methods and passing by vanishing points, optical flow, and stereoscopic based techniques. A novel segmentation approach for UAV attitude estimation based on polarization is proposed. Our future insightes for attitude estimation from uncalibrated catadioptric sensors are also discussed.     

改进PID的无人机飞行姿态角控制消颤算法

无人机在整个纵平面飞行过程中,由于飞行姿态角的大幅度变化以及气流的作用,导致机身颤抖,影响飞行稳定性.提出一种基于PID变结构控制的无人机飞行姿态角控制消颤算法,首先进行了无人机飞行姿态角控制系统的被控对象参量分析,构建无人机在姿态角变化剧烈、大迎角飞行时的三通道模型,采用变结构控制方法进行控制器设计.结合小扰动原理和Lyapunov稳定性原理进行扰动抑制和稳定性证明,采用梯度算法调整权值进行飞行姿态角控制的消颤处理,采用自适应算法在线调整权值实现PID变结构控制改进.仿真结果表明:采用该算法进行无人机飞行姿态角控制和消颤处理,大幅度提高无人机飞行定姿的精度,横滚角、俯仰角和航向角的控制精度有较大提高,稳定性和收敛性较好,确保了无人机飞行稳定性.     

基于气动参数调节的无人机抗扰动控制算法

无人机飞行受到气动阻尼扰动,从而导致控制稳定性不好。当前采用翼型截面气动参数调节的方法进行无人机抗扰控制,以扭角以及振动方向等参数为约束指标,参数调节的模糊度较大,对气动姿态参数调节的稳定性不好。文中提出基于气动参数调节的无人机抗扰动控制算法。该算法根据无人机的飞行工况构建各阶模态对应的气弹耦合方程,在速度坐标系、体坐标系、弹道坐标系三维坐标系下构建无人机的飞行动力学和运动学模型;采用卡尔曼滤波方法实现对无人机飞行参数的融合调节和小扰动抑制处理,并采用末端位置参考模型进行无人机飞行轨迹的空间规划设计;在卡尔曼滤波预估模型中实现对动力学模型的线性化处理,采用气弹模态参数识别方法进行无人机的飞行扰动调节;将姿态控制作为内环,获得位置环状态反馈调节参数;以无人机的升力系数和扭力系数作为气动惯性参数进行飞行姿态的稳定性调节,从而实现无人机抗扰动控制律的优化设计。采集飞机的俯仰角、横滚角和航向角作为原始数据在Matlab中进行仿真分析,仿真结果表明,采用所提方法进行无人机抗扰动控制的稳定性较好,对气动参数进行在线估计的准确性较高,航向角误差降低12.4%,抗扰动能力提升8dB,收敛时间比传统方法缩短0.14 s,无人机飞行的抗扰动性和飞行稳定性得到提高。所提方法在无人机飞行控制中具有很好的应用价值。     

Rectified catadioptric stereo sensors

It has been shown elsewhere how mirrors can be used to capture stereo images with a single camera, an approach termed catadioptric stereo. We present novel catadioptric sensors that use mirrors to produce rectified stereo images. The scanline correspondence of these images benefits real-time stereo by avoiding the computational cost and image degradation due to resampling when rectification is performed after image capture. First, we develop a theory which determines the number of mirrors that must be used and the constraints on those mirrors that must be satisfied to obtain rectified stereo images with a single camera. Then, we discuss in detail the use of both one and three mirrors. In addition, we show how the mirrors should be placed in order to minimize sensor size for a given baseline, an important design consideration. In order to understand the feasibility of building these sensors, we analyze rectification errors due to misplacement of the camera with respect to the mirrors     

Catadioptric Stereo Using Planar Mirrors

By using mirror reflections of a scene, stereo images can be captured with a single camera (catadioptric stereo). In addition to simplifying data acquisition single camera stereo provides both geometric and radiometric advantages over traditional two camera stereo. In this paper, we discuss the geometry and calibration of catadioptric stereo with two planar mirrors. In particular, we will show that the relative orientation of a catadioptric stereo rig is restricted to the class of planar motions thus reducing the number of external calibration parameters from 6 to 5. Next we derive the epipolar geometry for catadioptric stereo and show that it has 6 degrees of freedom rather than 7 for traditional stereo. Furthermore, we show how focal length can be recovered from a single catadioptric image solely from a set of stereo correspondences. To test the accuracy of the calibration we present a comparison to Tsai camera calibration and we measure the quality of Euclidean reconstruction. In addition, we will describe a real-time system which demonstrates the viability of stereo with mirrors as an alternative to traditional two camera stereo.     

Markov random fields for catadioptric image processing

Images obtained with catadioptric sensors contain significant deformations which prevent the direct use of classical image treatments. Thus, Markov random fields (MRF) whose usefulness is now obvious for projective image processing, cannot be used directly on catadioptric images because of the inadequacy of the neighborhood. In this paper, we propose to define a new neighborhood for MRF by using the equivalence theorem developed for central catadioptric sensors. We show the importance of this adaptation for segmentation, image restoration and motion detection.     

On improved calibration method for the catadioptric omnidirectional vision with a single viewpoint

The single viewpoint constraint is a principal optical characteristic for most catadioptric omnidirectional vision. Single viewpoint catadioptric omnidirectional vision is very useful because it allows the generation of geometrically correct perspective images from one omnidirectional image. Therefore precise calibration for single viewpoint constraint is needed during system assembling. However, in most image detection based calibration methods, the nonlinear optical distortion brought by lens is often neglected. Hence the calibration precision is poor. In this paper, a new calibration method of single viewpoint constraint for the catadioptric omni-directional vision is proposed. Firstly, an image correction algorithm is obtained by training a neural network. Then, according to characteristics of the space circular perspective projection, the corrected image of the mirror boundary is used to estimate its position and attitude relative to the camera to guide the calibration. Since the estimate is conducted based on actual imaging model rather than the simplified model, the estimate error is largely reduced, and the calibration accuracy is significantly improved. Experiments are conducted on simulated images and real images to show the accuracy and the effectiveness of the proposed method.     

Calibration of Central Catadioptric Cameras Using a DLT-Like Approach

In this study, we present a calibration technique that is valid for all single-viewpoint catadioptric cameras. We are able to represent the projection of 3D points on a catadioptric image linearly with a 6×10 projection matrix, which uses lifted coordinates for image and 3D points. This projection matrix can be computed from 3D–2D correspondences (minimum 20 points distributed in three different planes). We show how to decompose it to obtain intrinsic and extrinsic parameters. Moreover, we use this parameter estimation followed by a non-linear optimization to calibrate various types of cameras. Our results are based on the sphere camera model which considers that every central catadioptric system can be modeled using two projections, one from 3D points to a unitary sphere and then a perspective projection from the sphere to the image plane. We test our method both with simulations and real images, and we analyze the results performing a 3D reconstruction from two omnidirectional images.     

一种反射折射摄像机的简易标定方法

Central catadioptric cameras are widely used in virtual reality and robot navigation, and the camera calibration is a prerequisite for these applications. In this paper, we propose an easy calibration method for central catadioptric cameras with a 2D calibration pattern. Firstly, the bounding ellipse of the catadioptric image and field of view (FOV) are used to obtain the initial estimation of the intrinsic parameters. Then, the explicit relationship between the central catadioptric and the pinhole model is used to initialize the extrinsic parameters. Finally, the intrinsic and extrinsic parameters are refined by nonlinear optimization. The proposed method does not need any fitting of partial visible conic, and the projected images of 2D calibration pattern can easily cover the whole image, so our method is easy and robust. Experiments with simulated data as well as real images show the satisfactory performance of our proposed calibration method.     

A calibration method for paracatadioptric camera from sphere images

For paracatadioptric camera, the estimation of intrinsic parameters from sphere images is still an open and challenging problem. In this paper, we propose a calibration method for paracatadioptric camera based on sphere images, which only requires that the projected contour of parabolic mirror is visible on the image plane in one view. We have found that, under central catadioptric camera, a sphere is projected to two conics on the image plane, which are defined as a pair of antipodal sphere images. The conic that is visible on the image plane is called the sphere image, while the other invisible conic is called the antipodal sphere image. In the other aspect, according to the image formation of central catadioptric camera, these two conics can also be considered as the projections of two parallel circles on the viewing sphere by a virtue camera. That is to say, if three pairs of antipodal sphere images are known, central catadioptric camera can be directly calibrated by the calibration method based on two parallel circles. Therefore, the problem of calibrating central catadioptric camera is transferred to the estimations of sphere images and their antipodal sphere images. Based on this idea, we first initialize the intrinsic parameters of the camera by the projected contour of parabolic mirror, and use them to initialize the antipodal sphere images. Next, we study properties of several pairs of antipodal sphere images under paracatadioptric camera. Then, these properties are used to optimize sphere images and their antipodal sphere images, so as to calibrate the paracatadioptric camera. Experimental results on both simulated and real image data have demonstrated the effectiveness of our method.     

一种反射折射摄像机的简易标定方法

Central catadioptric cameras are widely used in virtual reality and robot navigation,and the camera calibration is a prerequisite for these applications.In this paper,we propose an easy calibration method for central catadioptric cameras with a 2D calibration pattern.Firstly,the bounding ellipse of the catadioptric image and field of view (FOV) are used to obtain the initial estimation of the intrinsic parameters.Then,the explicit relationship between the central catadioptric and the pinhole model is used to initialize the extrinsic parameters.Finally,the intrinsic and extrinsic parameters are refined by nonlinear optimization.The proposed method does not need any fitting of partial visible conic,and the projected images of 2D calibration pattern can easily cover the whole image,so our method is easy and robust.Experiments with simulated data as well as real images show the satisfactory performance of our proposed calibration method.     

A Theory of Single-Viewpoint Catadioptric Image Formation

Conventional video cameras have limited fields of view which make them restrictive for certain applications in computational vision. A catadioptric sensor uses a combination of lenses and mirrors placed in a carefully arranged configuration to capture a much wider field of view. One important design goal for catadioptric sensors is choosing the shapes of the mirrors in a way that ensures that the complete catadioptric system has a single effective viewpoint. The reason a single viewpoint is so desirable is that it is a requirement for the generation of pure perspective images from the sensed images. In this paper, we derive the complete class of single-lens single-mirror catadioptric sensors that have a single viewpoint. We describe all of the solutions in detail, including the degenerate ones, with reference to many of the catadioptric systems that have been proposed in the literature. In addition, we derive a simple expression for the spatial resolution of a catadioptric sensor in terms of the resolution of the cameras used to construct it. Moreover, we include detailed analysis of the defocus blur caused by the use of a curved mirror in a catadioptric sensor.     

基于深度学习的四旋翼无人机控制系统设计

针对传统四旋翼无人机控制系统受到外界干扰,无法及时躲避障碍物而导致控制精准度低的问题,提出了基于深度学习的四旋翼无人机控制系统设计。根据四旋翼无人机控制系统总体结构,加入超声波测距模块。依据系统硬件框图,采用TMS320F28335型号主控芯片,实现关键态势智能分析。以串级 PID 控制器的控制对象为无人机姿态角度,控制电机转速。根据DSP发出不同占空比的PWM信号,改变无人机飞行姿态,依据执行机构驱动原理,保证无人机飞行时的平衡状态。使用红外遥控系统,应用编/解码操控集成电路芯片,采用TS0P1738型号红外线接收器,适合于红外线遥控数据传输。构建深度学习目标控制模型,利用处突阵法与三角形相似原理,计算像素尺寸,获取障碍物距无人机当前位置距离,避免受到外界障碍物干扰。自适应扩展Kalman滤波器技术对无人机自动控制系统有效减小测量误差,准确地对机动目标进行追踪。由系统调试结果可知,该系统控制的俯仰角、航向角、横滚角与实际值一致,对处理突发性群体事件具有重要意义。     

多标记室内小型无人机定位与姿态估计方法

随着无人机（Unmanned Aerial Vehicle,UAV）小型化、轻便化的发展,因其价格低廉,以及在娱乐和服务领域的广泛使用的特点,使得如何实现一个便捷且易实现的自主飞行跟踪系统成为关注点。由于无人机在室内GPS信号弱,使得跟踪与姿态获取成为进一步室内无人机自主控制的重点与难点。与动辄几十万美元搭建的无人机跟踪系统相比,采用低成本单目摄像机的无人机跟踪系统具有更高的科研价值和更广泛的应用前景。针对目前流行的基于增强现实（Augmented Reality,AR）技术的ArUco标记算法和颜色空间域标记算法,设计了一种多标记的无人机跟踪系统。在无人机目标跟踪过程中比较两种方法,验证了两种方法非接触式深度传感器无人机跟踪和姿态估计的效果,并比较了两种方法对空间亮度与空间颜色复杂度的鲁棒性,以及不同跟踪距离下视频中无人机检出率与跟踪精度。实验结果表明,基于深度摄像机获得的无人机位置和姿态数据,无人机可以进行自主的PID控制飞行,且AR标记在复杂环境下无人机的检出率、跟踪实时性、姿态估计精度以及鲁棒性都优于颜色标记,为之后室内无人机在非接触式传感的控制、路径规划、自主规避等进一步实验研究提供了无人机的位置和姿态数据。     

Identical Projective Geometric Properties of Central Catadioptric Line Images and Sphere Images with Applications to Calibration

Central catadioptric cameras are imaging devices that use mirrors to enhance the field of view while preserving a single effective viewpoint. Lines and spheres in space are all projected into conics in the central catadioptric image planes, and such conics are called line images and sphere images, respectively. We discovered that there exists an imaginary conic in the central catadioptric image planes, defined as the modified image of the absolute conic (MIAC), and by utilizing the MIAC, the novel identical projective geometric properties of line images and sphere images may be exploited: Each line image or each sphere image is double-contact with the MIAC, which is an analogy of the discovery in pinhole camera that the image of the absolute conic (IAC) is double-contact with sphere images. Note that the IAC also exists in the central catadioptric image plane, but it does not have the double-contact properties with line images or sphere images. This is the main reason to propose the MIAC. From these geometric properties with the MIAC, two linear calibration methods for central catadioptric cameras using sphere images as well as using line images are proposed in the same framework. Note that there are many linear approaches to central catadioptric camera calibration using line images. It seems that to use the properties that line images are tangent to the MIAC only leads to an alternative geometric construction for calibration. However, for sphere images, there are only some nonlinear calibration methods in literature. Therefore, to propose linear methods for sphere images may be the main contribution of this paper. Our new algorithms have been tested in extensive experiments with respect to noise sensitivity.     

Cooperative Control of Multiple UAVs for Source Seeking

This paper considers the problem of unknown scalar field source seeking using multiple UAVs subject to input constraints. In this problem, each UAV can only measure the scalar field value at its current location. In order to seek the scalar field source, cooperation of multiple UAVs is carried out by adopting a leader-follower formation strategy. A least squares method is introduced to estimate the gradient of the scalar field at the leader UAV location based on the measurements of all UAVs. By using the estimated gradient, this paper proposes a guidance law for the heading of the leader UAV, and a sliding mode based heading rate controller is designed for the leader UAV to follow the desired heading angle. Furthermore, a heading rate controller is developed for each follower UAV to achieve circular formation around the leader UAV. Finally, simulation results are provided to demonstrate the effectiveness of the proposed approach.     

小型四旋翼无人机序列影像三维重建实验

传统航空影像三维重建受到时效性、成本等限制,而无人机三维重建技术是解决中小范围三维地形问题的有效手段。基于小型四旋翼无人机航拍序列影像开展了三维重建实验。首先分析了小型无人机航拍三维重建流程的特点,随后重点讨论了无人机三维重建中涉及的关键问题,包括无人机飞行姿态控制、航速、航高、重叠度和拍照模式选择等,最后对比了不同软件的重建结果。实验结果表明,通过优化控制无人机飞行姿态、航高、航速、重叠度等因素,选择合理的序列影像获取模式,在不同的软件中均可完成精度较高的精细地物三维重建。