基于多尺度变形卷积的特征金字塔光流计算方法

针对现有深度学习光流计算方法的运动边缘模糊问题,提出了一种基于多尺度变形卷积的特征金字塔光流计算方法.首先,构造基于多尺度变形卷积的特征提取模型,显著提高图像边缘区域特征提取的准确性;然后,将多尺度变形卷积特征提取模型与特征金字塔光流计算网络耦合,提出一种基于多尺度变形卷积的特征金字塔光流计算模型;最后,设计一种结合图像与运动边缘约束的混合损失函数,通过指导模型学习更加精准的边缘信息,克服了光流计算运动边缘模糊问题.分别采用MPI-Sintel和KITTI2015测试图像集对该方法与代表性的深度学习光流计算方法进行综合对比分析.实验结果表明,该方法具有更高的光流计算精度,有效解决了光流计算的边缘模糊问题.     

图像光流联合驱动的变分光流计算新方法

提出一种基于图像光流联合驱动的变分光流计算方法。数据项采用灰度守恒和梯度守恒相结合、局部约束与全局约束结合的思想,并引入正则化因子提高计算精度。平滑项采用图像与光流联合驱动的各向异性平滑策略,将数据项与平滑项紧密地联系起来,并通过设计扩散张量的两个本征值来控制光流扩散速度。最后采用多分辨率分层细化策略解决大位移问题。实验结果证明,该计算模型在背景复杂、光照变化、运动边界等情况的光流计算具有很好的效果。     

基于3D结构张量模型的稠密光流实现

像素的3D结构张量能够很好的表达视频帧序列之间的局部运动模型,因此,稠密光流的估计可以使用3D结构张量。根据光流的亮度恒定假设,光流估计问题可以转化为计算结构张量的特征向量,而不是复杂的线性系统计算。在一个有着多运动模型的场景中,有多个全局3D结构张量模型,每个模型对应为一个高斯分布,模型是在线更新的。为了提高效果,需要进行由粗到细的迭代过程。实验显示3D结构张量能够良好的表达运动模型。     

基于运动优化语义分割的变分光流计算方法

针对光照变化和大位移运动等复杂场景下图像序列变分光流计算的边缘模糊与过度分割问题,文中提出基于运动优化语义分割的变分光流计算方法.首先,根据图像局部区域的去均值归一化匹配模型,构建变分光流计算能量泛函.然后,利用去均值归一化互相关光流估计结果,获取图像运动边界信息,优化语义分割,设计运动约束语义分割的变分光流计算模型.最后,融合图像不同标签区域光流,获得光流计算结果.在Middlebury、UCF101数据库上的实验表明,文中方法的光流估计精度与鲁棒性较高,尤其对光照变化、弱纹理和大位移运动等复杂场景的边缘保护效果较优.     

改进非线性结构张量的含噪图像边缘检测

现有的边缘检测方法在含噪图像中的检测性能不佳.针对含噪图像的边缘检测问题,提出了利用引导核改进基于非线性结构张量的含噪图像边缘检测方法.首先,计算含噪图像的张量积.然后,根据图像梯度对张量积进行扩散,图像梯度依赖张量积本身.扩散方程中的扩散矩阵包含张量积,该张量积是通过各向异性的引导核进行空间自适应平均,而不是通过各向同性的高斯核进行平均.最后计算扩散张量积的特征值和特征向量,并基于此检测图像的边缘.将所提方法与基于线性结构张量的边缘检测方法、基于张量梯度扩散的非线性结构张量的边缘检测方法、基于图像梯度扩散的非线性结构张量的边缘检测方法进行比较,实验结果表明,所提方法可以得到更为清晰的边缘,并且检测结果中噪声较少.     

结合高斯混合模型与多通道双边滤波的RGBD场景流计算方法

针对现有RGBD场景流计算方法在大位移、运动遮挡等复杂运动场景中存在计算准确性与可靠性较低的问题，文中提出结合高斯混合模型与多通道双边滤波的RGBD场景流计算方法.首先，构造基于高斯混合模型的光流聚类分割模型，从光流中提取目标运动信息，逐层优化深度图分层分割结果，获取高置信度的深度运动分层分割信息.然后，在场景流计算中引入多通道双边滤波优化，建立结合高斯混合模型与多通道双边滤波的RGBD场景流计算模型，克服场景流计算边缘模糊问题.最后，在Middlebury、MPI-Sintel数据集上的实验表明，文中方法在大位移、运动遮挡等复杂运动场景下具有较高的场景流计算准确性和鲁棒性，特别在边缘区域具有较好的保护效果.     

基于深度匹配的由稀疏到稠密大位移运动光流估计

针对非刚性大位移运动场景的光流计算准确性与鲁棒性问题, 提出一种基于深度匹配的由稀疏到稠密大位移运动光流估计方法. 首先利用深度匹配模型计算图像序列相邻帧的初始稀疏运动场; 其次采用网格化邻域支持优化模型筛选具有较高置信度的图像网格和匹配像素点, 获得鲁棒的稀疏运动场; 然后对稀疏运动场进行边缘保护稠密插值, 并设计全局能量泛函优化求解稠密光流场. 最后分别利用MPI-Sintel和KITTI数据库提供的测试图像集对本文方法和Classic + NL, DeepFlow, EpicFlow以及FlowNetS等变分模型、匹配策略和深度学习光流计算方法进行综合对比与分析, 实验结果表明本文方法相对于其他方法具有更高的光流计算精度, 尤其在非刚性大位移和运动遮挡区域具有更好的鲁棒性与可靠性.     

非局部加权邻域三角滤波TV-L1光流估计

目的 针对非刚性运动、运动遮挡与间断、大位移以及复杂边缘结构等困难场景图像序列光流计算的准确性与鲁棒性问题,提出一种基于加权邻域三角滤波的非局部TV-L¹光流计算方法。方法 首先设计非平方惩罚函数L¹模型与梯度守恒假设相结合的数据项,然后引入基于L¹模型与基于图像梯度自适应变化权重相结合的平滑项,并根据提出的鲁棒数据项与图像-光流联合控制平滑项建立TV-L¹光流计算能量函数模型。最后采用基于加权邻域三角的非局部约束项,通过引入图像金字塔分层变形计算策略,在每层图像光流计算时对光流计算结果进行基于加权邻域三角网格的中值滤波优化,提出基于加权邻域三角滤波的非局部TV-L¹光流计算模型。结果 分别采用MPI与Middlebury数据库测试图像序列对本文方法和LDOF、CLG-TV、SOF、Classic+NL等代表方法进行实验对比。本文方法光流计算结果的平均角误差（AAE）和平均端点误差（AEE）相对其他对比方法平均下降28.45%和28.42%,时间消耗相对传统方法增长5.16%。结论 相对于传统的光流计算方法,本文方法针对非刚体运动、运动遮挡与间断、大位移运动以及复杂边缘等困难场景具有较好的适用性,光流估计结果具有较高的精度和较好的鲁棒性。     

基于局部–全局建模与视觉相似引导的光流估计方法

光流估计是计算机视觉的核心任务.近年来,基于卷积神经网络的光流估计方法已取得很大成功,然而由于现有模型的卷积感受野有限,难以建模远距离的依赖关系,导致在大位移和局部歧义性区域的光流估计效果较差.此外,现有方法在光流上采样过程采用的插值操作会导致误差的传播放大,进而引起光流估计的运动边缘模糊等问题.针对以上问题,本文提出了一种基于局部–全局建模与视觉相似引导上采样的光流估计方法.首先,引入一个高效且简单的自注意力机制加强光流计算网络的局部和全局建模能力.通过提取更具有表达力的图像特征,降低因大位移和局部歧义性导致的光流估计误差问题.其次,基于物体视觉特征越相似,运动也越相似的假设,构建视觉相似引导的光流上采样网络模型.将特征的视觉相似性转化为运动的相似性进而指导光流上采样过程,提高了运动边界区域光流估计的精度.最后,分别采用MPI-Sintel和KITTI数据库测试图像集对本文方法和最先进的深度学习光流计算方法进行综合对比分析.实验结果表明,本文方法在所有对比方法中取得了最优的光流计算结果,尤其在大位移和运动边界区域显著提升了光流计算的精度.     

基于形态学算子的各向异性扩散去噪方法

针对相干增强扩散模型采用高斯线性滤波做图像预处理的不足,及扩散张量特征值的选取不适合平坦区域的去噪,易在平坦区域产生虚假边缘,文中提出了一个基于形态学算子的各向异性扩散去噪方法.该方法首先利用形态学闭开算子代替高斯滤波做预处理,然后结合二阶方向导数设计结构张量,且依据自适应的梯度阈值设计扩散张量的特征值.数值实验结果表明,改进后的方法在有效去除噪声的同时,还能很好地保持图像的细节特征和消除平坦区域的虚假边缘     

图像引导的二阶总广义变分稀疏深度图的稠密重构

利用图像颜色信息进行深度图重构,可以恢复对象边界处的深度不连续性,但无法保证对象内部的深度均匀性。为解决该问题,提出图像引导下总广义变分正则化的深度图重构模型。该模型利用扩散张量将图像提供的边缘信息引入二阶总广义变分正则项,使得重构深度在保持对象边缘的同时逼近分段仿射平面,从而保证恢复深度既保持对象边界处的不连续性,又具有对象内部的均匀性。通过Legendre-Fenchel变换将模型转换成等效的凸凹鞍点问题,从而得到高效的一阶原始对偶求解算法。实验结果表明,该方法能够恢复尖锐的对象边缘,同时保持对象内部的深度均匀性。与现有算法相比,所提方法具有更高的峰值信噪比、归一化互协方差和更低的平均绝对误差。     

Improving the robustness of variational optical flow through tensor voting

Differential optical flow methods allow the estimation of optical flow fields based on the first-order and even higher-order spatio-temporal derivatives (gradients) of sequences of input images. If the input images are noisy, for instance because of the limited quality of the capturing devices or due to poor illumination conditions, the use of partial derivatives will amplify that noise and thus end up affecting the accuracy of the computed flow fields. The typical approach in order to reduce that noise consists of smoothing the required gradient images with Gaussian filters, for instance by applying structure tensors. However, that filtering is isotropic and tends to blur the discontinuities that may be present in the original images, thus likely leading to an undesired loss of accuracy in the resulting flow fields. This paper proposes the use of tensor voting as an alternative to Gaussian filtering, and shows that the discontinuity preserving capabilities of the former yield more robust and accurate results. In particular, a state-of-the-art variational optical flow method has been adapted in order to utilize a tensor voting filtering approach. The proposed technique has been tested upon different datasets of both synthetic and real image sequences, and compared to both well known and state-of-the-art differential optical flow methods.     

Motion estimation and segmentation

In the general structure-from-motion (SFM) problem involving several moving objects in a scene, the essential first step is to segment moving objects independently. We attempt to deal with the problem of optical flow estimation and motion segmentation over a pair of images. We apply a mean field technique to determine optical flow and motion boundaries and present a deterministic algorithm. Since motion discontinuities represented by line process are embedded in the estimation of the optical flow, our algorithm provides accurate estimates of optical flow especially along motion boundaries and handles occlusion and multiple motions. We show that the proposed algorithm outperforms other well-known algorithms in terms of estimation accuracy and timing.     

Optical-flow based on an edge-avoidance procedure

This paper presents a differential optical flow method which accounts for two typical motion-estimation problems: (1) flow regularization within regions of uniform motion while (2) preserving sharp edges near motion discontinuities i.e., where motion is multimodal by nature. The method proposed is a modified version of the well known Lucas–Kanade (LK) algorithm. While many edge-preserving strategies try to minimize the effect of outliers by using a line process or a robust function, our method takes a novel approach to solve the problem. Based on documented assumptions, our method computes motion with a classical least-squares fit on a local neighborhood shifted away from where motion is likely to be multimodal. In this way, the inherent bias due to multiple motion around moving edges is avoided instead of being compensated. This edge-avoidance procedure is based on the non-parametric mean-shift algorithm which shifts the LK integration window away from local sharp edges. Our method also locally regularizes motion by performing a fusion of local motion estimates. The regularization is made with a covariance filter which minimizes the effect of uncertainties due in part to noise and/or lack of texture. Our method is compared with other edge-preserving methods on image sequences representing different challenges.     

A Database and Evaluation Methodology for Optical Flow

The quantitative evaluation of optical flow algorithms by Barron et al. (1994) led to significant advances in performance. The challenges for optical flow algorithms today go beyond the datasets and evaluation methods proposed in that paper. Instead, they center on problems associated with complex natural scenes, including nonrigid motion, real sensor noise, and motion discontinuities. We propose a new set of benchmarks and evaluation methods for the next generation of optical flow algorithms. To that end, we contribute four types of data to test different aspects of optical flow algorithms: (1) sequences with nonrigid motion where the ground-truth flow is determined by tracking hidden fluorescent texture, (2) realistic synthetic sequences, (3) high frame-rate video used to study interpolation error, and (4) modified stereo sequences of static scenes. In addition to the average angular error used by Barron et al., we compute the absolute flow endpoint error, measures for frame interpolation error, improved statistics, and results at motion discontinuities and in textureless regions. In October 2007, we published the performance of several well-known methods on a preliminary version of our data to establish the current state of the art. We also made the data freely available on the web at . Subsequently a number of researchers have uploaded their results to our website and published papers using the data. A significant improvement in performance has already been achieved. In this paper we analyze the results obtained to date and draw a large number of conclusions from them.     

Rigid body segmentation and shape description from dense opticalflow under weak perspective

We present an algorithm for identifying and tracking independently moving rigid objects from optical flow. Some previous attempts at segmentation via optical flow have focused on finding discontinuities in the flow field. While discontinuities do indicate a change in scene depth, they do not in general signal a boundary between two separate objects. The proposed method uses the fact that each independently moving object has a unique epipolar constraint associated with its motion. Thus motion discontinuities based on self-occlusion can be distinguished from those due to separate objects. The use of epipolar geometry allows for the determination of individual motion parameters for each object as well as the recovery of relative depth for each point on the object. The algorithm assumes an affine camera where perspective effects are limited to changes in overall scale. No camera calibration parameters are required. A Kalman filter based approach is used for tracking motion parameters with time     

Accurate optical flow computation under non-uniform brightness variations

In this paper, we present a very accurate algorithm for computing optical flow with non-uniform brightness variations. The proposed algorithm is based on a generalized dynamic image model (GDIM) in conjunction with a regularization framework to cope with the problem of non-uniform brightness variations. To alleviate flow constraint errors due to image aliasing and noise, we employ a reweighted least-squares method to suppress unreliable flow constraints, thus leading to robust estimation of optical flow. In addition, a dynamic smoothness adjustment scheme is proposed to efficiently suppress the smoothness constraint in the vicinity of the motion and brightness variation discontinuities, thereby preserving motion boundaries. We also employ a constraint refinement scheme, which aims at reducing the approximation errors in the first-order differential flow equation, to refine the optical flow estimation especially for large image motions. To efficiently minimize the resulting energy function for optical flow computation, we utilize an incomplete Cholesky preconditioned conjugate gradient algorithm to solve the large linear system. Experimental results on some synthetic and real image sequences show that the proposed algorithm compares favorably to most existing techniques reported in literature in terms of accuracy in optical flow computation with 100% density.     

On the Fourier Properties of Discontinuous Motion

Retinal image motion and optical flow as its approximation are fundamental concepts in the field of vision, perceptual and computational. However, the computation of optical flow remains a challenging problem as image motion includes discontinuities and multiple values mostly due to scene geometry, surface translucency and various photometric effects such as reflectance. In this contribution, we analyze image motion in the frequency space with respect to motion discontinuities and translucence. We derive the frequency structure of motion discontinuities due to occlusion and we demonstrate its various geometrical properties. The aperture problem is investigated and we show that the information content of an occlusion almost always disambiguates the velocity of an occluding signal suffering from the aperture problem. In addition, the theoretical framework can describe the exact frequency structure of Non-Fourier motion and bridges the gap between Non-Fourier visual phenomena and their understanding in the frequency domain.     

Non Uniform Multiresolution Method for Optical Flow and Phase Portrait Models: Environmental Applications

In this paper we define a complete framework for processing large image sequences for a global monitoring of short range oceanographic and atmospheric processes. This framework is based on the use of a non quadratic regularization technique for optical flow computation that preserves flow discontinuities. We also show that using an appropriate tessellation of the image according to an estimate of the motion field can improve optical flow accuracy and yields more reliable flows. This method defines a non uniform multiresolution approach for coarse to fine grid generation. It allows to locally increase the resolution of the grid according to the studied problem. Each added node refines the grid in a region of interest and increases the numerical accuracy of the solution in this region. We make use of such a method for solving the optical flow equation with a non quadratic regularization scheme allowing the computation of optical flow field while preserving its discontinuities. The second part of the paper deals with the interpretation of the obtained displacement field. For this purpose a phase portrait model used along with a new formulation of the approximation of an oriented flow field allowing to consider arbitrary polynomial phase portrait models for characterizing salient flow features. This new framework is used for processing oceanographic and atmospheric image sequences and presents an alternative to complex physical modeling techniques.