模糊C-均值聚类引导的Kinect深度图像修复算法

针对Kinect传感器所采集的深度图像中存在大面积空洞的问题,提出了一种模糊C-均值聚类引导的深度图像修复算法。该算法将同步获取的彩色图像和深度图像作为输入;利用模糊C-均值聚类算法对彩色图像进行聚类,聚类结果作为引导图像;然后对每个深度图像中的大面积空洞区域,利用改进的快速行进算法,从空洞边缘向空洞内部逐层修复空洞区域;最后,利用改进的双边滤波算法去除图像中的散粒噪声。实验表明该算法能有效修复Kinect深度图像中的空洞,修复后的图像在平滑度和边缘强度上优于传统算法。     

Depth map inpainting via sparse distortion model

The depth map captured from a real scene by the Kinect motion sensor is always influenced by noise and other environmental factors. As a result, some depth information is missing from the map. This distortion of the depth map directly deteriorates the quality of the virtual viewpoints rendered in 3D video systems. We propose a depth map inpainting algorithm based on a sparse distortion model. First, we train the sparse distortion model using the distortion and real depth maps to obtain two learning dictionaries: one for distortion and one for real depth maps. Second, the sparse coefficients of the distortion and the real depth maps are calculated by orthogonal matching pursuit. We obtain the approximate features of the distortion from the relationship between the learning dictionary and the sparse coefficients of the distortion map. The noisy images are filtered by the joint space structure filter, and the extraction factor is obtained from the resulting image by the extraction factor judgment method. Finally, we combine the learning dictionary and sparse coefficients from the real depth map with the extraction factor to repair the distortion in the depth map. A quality evaluation method is proposed for the original real depth maps with missing pixels. The proposed method achieves better results than comparable methods in terms of depth inpainting and the subjective quality of the rendered virtual viewpoints.     

Automatic inpainting by removing fence-like structures in RGBD images

Recent inpainting techniques usually require human interactions which are labor intensive and dependent on the user experiences. In this paper, we introduce an automatic inpainting technique to remove undesired fence-like structures from images. Specifically, the proposed technique works on the RGBD images which have recently become cheaper and easier to obtain using the Microsoft Kinect. The basic idea is to segment and remove the undesired fence-like structures by using both depth and color information, and then adapt an existing inpainting algorithm to fill the holes resulting from the structure removal. We found that it is difficult to achieve a satisfactory segmentation of such structures by only using the depth channel. In this paper, we use the depth information to help identify a set of foreground and background strokes, with which we apply a graph-cut algorithm on the color channels to obtain a more accurate segmentation for inpainting. We demonstrate the effectiveness of the proposed technique by experiments on a set of Kinect images.     

基于不确定度评价的Kinect深度图预处理

针对应用在机器人三维(3D)场景感知测量中,Kinect深度图的联合双边滤波(JBF)存在降低原始场景深度信息精确度的制约性问题,提出一种新的预处理算法。首先,通过构建深度图的测量和采样模型,得到深度图的蒙特卡罗不确定度评价模型;其次,依据该模型计算得到深度值估计区间,实现噪声点与非噪声点的判定及滤除;最后,利用估计区间均值完成噪声点的修复。实验结果表明,该算法在噪声滤波的同时保证了非噪声的不变性;非噪声的不变性以及基于估计均值的噪声修复使原始深度梯度具有不变性;与联合彩色深度图的双边滤波相比,预处理结果图物体边缘轮廓清晰不变且其均方误差降低了15.25%~28.79%。因此,该预处理算法达到了提高三维场景深度信息精确度的目的。     

Guided depth enhancement via a fast marching method

Range imaging sensors such as Kinect and time-of-flight cameras can produce aligned depth and color images in real time. However, the depth maps captured by such sensors contain numerous invalid regions and suffer from heavy noise. These defects more or less influence the use of depth information in practical applications. In order to enhance the depth maps, this paper proposes a new inpainting approach based on the fast marching method (FMM). We extend the inpainting model and the propagation strategy of FMM to incorporate color information for depth inpainting. An edge-preserving guided filter is further applied for noise reduction. To validate our algorithm, we perform experiments on both Kinect data and Middlebury dataset which, respectively, provide qualitative and quantitative results. Meanwhile, we also compare it to the original FMM and other two state-of-the-art depth enhancement methods. Experimental results show that our method performs better than the local methods in terms of both visual and metric qualities, and it achieves visually comparable results to the time-consuming global method.     

A divide-and-conquer hole-filling method for handling disocclusion in single-view rendering

Large holes are unavoidably generated in depth image based rendering (DIBR) using a single color image and its associated depth map. Such holes are mainly caused by disocclusion, which occurs around the sharp depth discontinuities in the depth map. We propose a divide-and-conquer hole-filling method which refines the background depth pixels around the sharp depth discontinuities to address the disocclusion problem. Firstly, the disocclusion region is detected according to the degree of depth discontinuity, and the target area is marked as a binary mask. Then, the depth pixels located in the target area are modified by a linear interpolation process, whose pixel values decrease from the foreground depth value to the background depth value. Finally, in order to remove the isolated depth pixels, median filtering is adopted to refine the depth map. In these ways, disocclusion regions in the synthesized view are divided into several small holes after DIBR, and are easily filled by image inpainting. Experimental results demonstrate that the proposed method can effectively improve the quality of the synthesized view subjectively and objectively.     

基于目标特征的植株深度图像修复

针对植株深度图像的像素错误和缺失、常见的滤波方法无法准确修复植株深度图 像的问题,提出一种基于目标特征的植株深度图像修复方法。首先基于颜色和空间信息的图像 分割算法对植株彩色图像进行目标分割,再检索每个目标的外轮廓,并对外轮廓进行多边形拟 合;其次,基于目标区域搜索深度图像中具有正确深度值的像素作为目标区域采样点,并对叶 片区域的图像进行归一化;最后,利用空间拟合法计算各目标区域的方程,修复区域内小面积 错误和缺失的深度值,同时采用支持向量机和空间变换运算对大面积错误和缺失深度值的叶片 区域进行修复。实验结果表明,该方法能够准确地修复植株深度图像中错误、缺失的深度数据, 且能够有效地保护目标区域的边缘信息。     

Edge-guided GAN：边界信息引导的深度图像修复

目的 目前大多数深度图像修复方法可分为两类：色彩图像引导的方法和单个深度图像修复方法。色彩图像引导的方法利用色彩图像真值,或其上一帧、下一帧提供的信息来修复深度图像。若缺少相应信息,这类方法是无效的。单个深度图像修复方法可以修复数据缺失较少的深度图像。但是,无法修复带有孔洞（数据缺失较大）的深度图像。为解决以上问题,本文将生成对抗网络（generative adversarial network,GAN）应用于深度图像修复领域,提出了一种基于GAN的单个深度图像修复方法,即Edge-guided GAN。方法 首先,通过Canny算法获得待修复深度图像的边界图像,并将此两个单通道图像（待修复深度图像和边界图像）合并成一个2通道数据;其次,设计Edge-guided GAN高性能的生成器、判别器和损失函数,将此2通道数据作为生成器的输入,训练生成器,以生成器生成的深度图像（假值）和深度图像真值为判别器的输入,训练判别器;最终得到深度图像修复模型,完成深度图像修复。结果 在Apollo scape数据集上与其他4种常用的GAN、不带边界信息的Edge-guided GAN进行实验分析。在输入尺寸为256×256像素,掩膜尺寸为32×32像素情况下,Edge-guided GAN的峰值信噪比（peak signal-to-noise ratio,PSN）比性能第2的模型提高了15.76%;在掩膜尺寸为64×64像素情况下,Edge-guided GAN的PSNR比性能第2的模型提高了18.64%。结论 Edge-guided GAN以待修复深度图像的边界信息为其修复的约束条件,有效地提取了待修复深度图像特征,大幅度地提高了深度图像修复的精度。     

Depth map enhancement based on color and depth consistency

Current low-cost depth sensing techniques, such as Microsoft Kinect, still can achieve only limited precision. The resultant depth maps are often found to be noisy, misaligned with the color images, and even contain many large holes. These limitations make it difficult to be adopted by many graphics applications. In this paper, we propose a computational approach to address the problem. By fusing raw depth values with image color, edges and smooth priors in a Markov random field optimization framework, both misalignment and large holes can be eliminated effectively, our method thus can produce high-quality depth maps that are consistent with the color image. To achieve this, a confidence map is estimated for adaptive weighting of different cues, an image inpainting technique is introduced to handle large holes, and contrasts in the color image are also considered for an accurate alignment. Experimental results demonstrate the effectiveness of our method.     

基于Kinect深度信息的实时三维重建和滤波算法研究

分析了基于Kinect输出的深度数据进行场景实时三维重建的算法。针对实现过程中出现的深度图像噪声过大的问题,根据其信号结构的特点给出了改进的双边滤波算法。新算法利用已知的深度图像噪声范围,将权值函数修改为二值函数,并结合RGB图像弥补了缺失的深度信息。实验表明,新算法无论是在降噪性能还是计算效率上,都大大优于已有的双边滤波,其中计算速度是原始算法的6倍。     

基于Kinect传感器和ORB特征的视觉SLAM算法设计与实现

介绍了一个基于嵌入式平台和Kinect传感器的同时定位与地图创建算法的设计与实现。Kinect传感器包括一个可见光彩色摄像头和一个利用结构光测量深度的红外CMOS摄像头。 算法利用ORB算子作为环境特征点的描述信息,并利用基于边沿的最近邻修复方法对深度图像进行修正以获得完整的深度信息。在此基础上,利用LSH方法进行特征点的匹配。实验结果表明,基于ORB特征的视觉SLAM算法具有较好的实用性和良好的定位精度,可以广泛应用于室内机器人的自主导航任务。     

A novel hardware plane fitting implementation and applications for bionic vision

This paper presents a high-speed real-time plane fitting implementation on a field-programmable gate array (FPGA) platform. A novel hardware-based least squares algorithm fits planes to patches of points within a depth image captured using a Microsoft Kinect v2 sensor. The validity of a plane fit and the plane parameters are reported for each patch of 11 by 11 depth pixels. The high level of parallelism of operations in the algorithm has allowed for a fast, low-latency hardware implementation on an FPGA that is capable of processing depth data at a rate of 480 frames per second. A hybrid hardware–software end-to-end system integrates the hardware solution with the Kinect v2 sensor via a computer and PCI express communication link to a Terasic TR4 FPGA development board. We have also implemented two proof-of-concept object detection applications as future candidates for bionic vision systems. We show that our complete end-to-end system is capable of running at 60 frames per second. An analysis and characterisation of the Kinect v2 sensor errors has been performed in order to specify logic precision requirements, statistical testing of the validity of a plane fit, and achievable plane fitting angle resolution.     

基于双十字曲率驱动扩散模型的图像修复算法

当前各种基于曲率驱动扩散(CDD)模型的图像修复算法在修复待修复点时均只利用了其邻域中的4个点的参考信息,使修复后的图像边缘过渡不自然且修复精度不够高。针对以上问题,提出了基于双十字CDD的图像修复算法。该算法在充分利用原始CDD算法中4个邻域点的参考信息得到待修复点的修复像素值的基础上,再利用新引入的4个点的参考信息得到一个新的修复像素值,并将这两个修复像素值进行加权平均得到最终的修复像素值。最后,将提出的算法和原始的CDD算法以及改进的CDD算法用于实例验证,其结果表明,新提出的算法在不增加算法时间复杂度的条件下,使得图像边缘过渡更加自然,修复精度得到了有效提高。     

纹理边缘引导的深度图像超分辨率重建

目的 深度图像作为一种普遍的3维场景信息表达方式在立体视觉领域有着广泛的应用。Kinect深度相机能够实时获取场景的深度图像,但由于内部硬件的限制和外界因素的干扰,获取的深度图像存在分辨率低、边缘不准确的问题,无法满足实际应用的需要。为此提出了一种基于彩色图像边缘引导的Kinect深度图像超分辨率重建算法。方法 首先对深度图像进行初始化上采样,并提取初始化深度图像的边缘;进一步利用高分辨率彩色图像和深度图像的相似性,采用基于结构化学习的边缘检测方法提取深度图的正确边缘;最后找出初始化深度图的错误边缘和深度图正确边缘之间的不可靠区域,采用边缘对齐的策略对不可靠区域进行插值填充。结果 在NYU2数据集上进行实验,与8种最新的深度图像超分辨率重建算法作比较,用重建之后的深度图像和3维重建的点云效果进行验证。实验结果表明本文算法在提高深度图像的分辨率的同时,能有效修正上采样后深度图像的边缘,使深度边缘与纹理边缘对齐,也能抑制上采样算法带来的边缘模糊现象;3维点云效果显示,本文算法能准确区分场景中的前景和背景,应用于3维重建等应用能取得较其他算法更好的效果。结论 本文算法普遍适用于Kinect深度图像的超分辨率重建问题,该算法结合同场景彩色图像与深度图像的相似性,利用纹理边缘引导深度图像的超分辨率重建,可以得到较好的重建结果。     

基于深度信息的指尖追踪及手势识别

提出了一个基于深度信息对手指和手部进行实时跟踪,并可用于手势识别的方案。用Kinect获取深度信息,然后生成手部的三维点云,进行过滤转换成像素矩阵;使用K-curvature算法获取指尖和手掌方位,然后通过手指之间的相关距离进行手指标定。实验结果证明该方案识别追踪效果稳定且高效,不受光照和复杂背景影响,能够同时跟踪双手共10个手指和2个掌心的动作轨迹,并用于手势识别。     

An enhanced depth map based rendering method with directional depth filter and image inpainting

Depth image-based rendering (DIBR), which is used to render virtual views with a color image and the corresponding depth map, is one of the key techniques in the 2D to 3D video conversion process. In this paper, a novel method is proposed to partially solve two puzzles of DIBR, i.e. visual image generation and hole filling. The method combines two different approaches for synthesizing new views from an existing view and a corresponding depth map. Disoccluded parts of the synthesized image are first classified as either smooth or highly structured. At structured regions, inpainting is used to preserve the background structure. In other regions, an improved directional depth smoothing is used to avoid disocclusion. Thus, more details and straight line structures in the generated virtual image are preserved. The key contributions include an enhanced adaptive directional filter and a directional hole inpainting algorithm. Experiments show that the disocclusion is removed and the geometric distortion is reduced efficiently. The proposed method can generate more visually satisfactory results.     

面向图像修复的域相似算法

基于偏微分方程（PDE）的图像修复和基于纹理合成的图像修复是目前数字图像修复中的重要方法,虽然均能较好地修复图像,但是修复的效率较低。提出了一种采用域相似修复图像的新算法,先对待修复区域边界上的所有待修复点计算优先级,然后按照优先级从大到小的顺序修复图像;该算法以像素点邻域的相似来衡量两个像素点相似的程度,充分考虑了待修复像素的邻域中已知信息对该像素的影响。仿真实验结果表明,该算法不仅能较好地修复图像,而且在同等修复区域和修复效果的条件下具有更高的修复效率。     

基于深度信息的移动机器人室内环境三维地图创建

针对使用扩展卡尔曼滤波(EKF）进行环境地图的创建对线性系统效果较好而对非线性系统的线性化受误差影响较大的问题,提出一种基于对Kinect采集到的环境数据和迭代扩展卡尔曼滤波(IEKF）算法的室内环境三维地图创建。该方法使用成本较低的Kinect传感器获取深度数据然后结合IEKF实现摄像头轨迹预测,最后利用最近点迭代(ICP）算法对深度图像进行配准得到室内环境三维点云图。实验结果表明,IEKF算法与传统的EKF算法相比,得到的轨迹更平滑、误差更小,同时所得到的三维点云图更加光滑。该方法实现了三维地图构建,较为实用,效果较好。     

基于引导采样的Kinect深度图修补算法

针对Kinect采集到的深度图带有大量的结构性缺失,提出了一种新的深度图空洞噪声修补算法。算法首先将深度图所对应的彩色图片转化为灰度图,然后用Kmeans算法将彩色图转化而来的灰度图进行聚类处理,将生成的聚类图作为引导图。联合引导图对深度图空洞噪声边缘深度值采样,采集多个深度值并计算深度均值,最后使用深度均值来作为空洞的深度估计值。通过与MC-UE算法相比较,由于有引导图的矫正作用,边缘细节更加清晰准确。对于处理较小面积的空洞噪声,处理结果相较于MC-UE算法,均方误差仅降低4%左右。但对于处理较大面积的空洞噪声,均方误差较MC-UE算法降低了9.65%~14.32%。     

小波域的快速自适应图像修复算法

传统的基于偏微分方程的图像修复算法需要大量迭代,修复所耗时间较长,复杂度高。针对这一问题,提出了一种小波域的非迭代自适应图像修复算法。该算法对破损图像进行小波分解,找到待修复区域,根据待修复区域及其邻域像素值自适应选择修复模板大小,对修复模板内的像素值进行方向筛选,使修复过程严格按照等照度线方向行进,对修复后的图像进行小波重构。实验结果表明,该方法显著地缩短了修复时间,且对于图像的纹理细节、结构信息都达到了更好的修复效果。