Multi-scale gradient attention guidance and adaptive style fusion for image inpainting

Image inpainting aims to fill in the missing regions of damaged images with plausible content. Existing inpainting methods tend to produce ambiguous artifacts and implausible structures. To address the above issues, our method aims to fully utilize the information of known regions to provide style and structural guidance for missing regions. Specifically, the Adaptive Style Fusion (ASF) module reduces artifacts by transferring visual style features from known regions to missing regions. The Gradient Attention Guidance (GAG) module generates accurate structures by aggregating semantic information along gradient boundary regions. In addition, the Multi-scale Attentional Feature Extraction (MAFE) module extracts global contextual information and enhances the representation of image features. The sufficient experimental results on the three datasets demonstrate that our proposed method has superior performance in terms of visual plausibility and structural consistency compared to state-of-the-art inpainting methods.     

Image segmentation and inpainting using hierarchical level set and texture mapping

Image inpainting is an artistic procedure to recover a damaged painting or picture. We propose a novel approach for image inpainting by using the Mumford-Shah (MS) model and the level set method to estimate image structure of the damaged regions. This approach has been successfully used in image segmentation problem. Compared to some other inpainting methods, the MS model approach detects and preserves edges in the inpainting areas. We propose a fast and efficient algorithm that achieves both inpainting and segmentation. In previous works on the MS model, only one or two level set functions are used to segment an image. While this approach works well on simple cases, detailed edges cannot be detected in complicated image structures. Although multi-level set functions can be used to segment an image into many regions, the traditional approach causes extensive computations and the solutions depend on the location of initial curves. Our proposed approach utilizes faster hierarchical level set method and guarantees convergence independent of initial conditions. Because we detect both the main structure and the detailed edges, our approach preserves edges in the inpainting area. Also, exemplar-based approach for filling textured regions is employed. Experimental results demonstrate the advantage of our method.     

Generative image inpainting via edge structure and color aware fusion

Very recently, with the widespread research of deep learning, its achievements are increasingly evident in image inpainting tasks. However, many existing methods fail to effectively reconstruct vivid contents and refine structures. In order to solve this issue, in this paper, a novel two-stage generative adversarial network based on the fusion of edge structures and color aware maps is proposed. In the first-stage network, edges with missing regions are employed to train an edge structure generator. Meanwhile, the input image with missing regions is transformed into a global color feature map after the content aware fill algorithm and a large kernel size Gaussian filtering. In the second-stage network, the image fused from the edge map and the color map is used as a label to guide the network to reconstruct the refined image. Qualitative and quantitative experiments conducted on multiple public datasets demonstrate that the method proposed in this paper has superior performance.     

Hybrid error concealment method combining exemplar-based image inpainting and spatial interpolation

This paper proposes an efficient error concealment method for the reconstruction of pixels that are lost in video communication. The proposed method is developed by combining exemplar-based image inpainting for patch reconstruction and spatial interpolation for pixel reconstruction using adaptive threshold by local complexity. By exemplar-based image inpainting, regions with regular structures are reconstructed. For complex regions with irregular structures, just one pixel is reconstructed using the proposed spatial interpolation method. The proposed spatial interpolation method performs reconstruction by selecting adaptively directional interpolation or neighbor interpolation based on gradient information. Simulation results show that the proposed hybrid method performs reconstruction with significantly improved subjective quality compared with the previous spatial error concealment and image inpainting methods. The proposed method also gives substantial improvements of PSNR compared with the previous methods.     

FFTI: Image inpainting algorithm via features fusion and two-steps inpainting

In view of the faultiness that the existing image inpainting methods fail to make full use of the complete region to predict the missing region features when the object features are seriously missing, resulting in discontinuous features and fuzzy detail texture of the inpainting results, a fine inpainting method of incomplete image based on features fusion and two-steps inpainting (FFTI) is proposed in this paper. Firstly, the dynamic memory networks (DMN+) are used to fuse the external features and internal features of the incomplete image to generate the incomplete image optimization map. Secondly, a generation countermeasure generative network with gradient penalty constraints is constructed to guide the generator to rough repair the optimized incomplete image and obtain the rough repair map of the target to be repaired. Finally, the coarse repair graph is further optimized by the idea of coherence of relevant features to obtain the final fine repair graph. It is verified by simulation on three image data sets with different complexity, and compared with the existing dominant repair model for visual effect and objective data. The experimental results show that the results of the model repair in this paper are more reasonable in texture structure, better than other models in visual effect and objective data, and the Peak Signal-to-Noise Ratio of the proposed algorithm in the most challenging underwater targe dataset is 27.01, the highest Structural Similarity Index is 0.949.     

Identification of inpainted images and natural images for digital forensics

Image forensics is a form of image analysis for finding out the condition of an image in the complete absence of any digital watermark or signature.It can be used to authenticate digital images and identify their sources.While the technology of exemplar-based inpainting provides an approach to remove objects from an image and play visual tricks.In this paper,as a first attempt,a method based on zero-connectivity feature and fuzzy membership is proposed to discriminate natural images from inpainted images.Firstly,zero-connectivity labeling is applied on block pairs to yield matching degree feature of all blocks in the region of suspicious,then the fuzzy memberships are computed and the tampered regions are identified by a cut set.Experimental results demonstrate the effectiveness of our method in detecting inpainted images.     

采用图像修复的基于深度图像复制

在传统的基于深度图像复制(DIBR)的基础上提出一种基于图像修复的DIBR方法,将预处理深度图像和图像修复算法相结合来填补三维图像映射后的空洞。与传统方法相比更加灵活,本文方法仅需传输一路参考图像序列,从而有效降低DIBR系统的传输带宽。实验结果证明,本文所提出方法是有效的。     

Inpainting of binary images using the Cahn-Hilliard equation.

Image inpainting is the filling in of missing or damaged regions of images using information from surrounding areas. We outline here the use of a model for binary inpainting based on the Cahn-Hilliard equation, which allows for fast, efficient inpainting of degraded text, as well as super-resolution of high contrast images.     

基于三通道多小波紧标架的图像曲率修复模型

为解决正交小波域图像修复方法的现存小波 系数不能提供丢 失小波系数的充足信息问题,将曲率修复(CDD)模型扩展到非正交小波域,提出一种新的基 于三通道多小波 紧标架的图像CDD模型。进一步考虑到CDD模型的等照度线是按照直线连接的问 题,提出了一种新的CCD模型。新模型的修复在两个方向上进行,在法向上按照改进的曲率 函数进行修复, 在切向上实现输运机制。新模型结合了多小波紧标架分解技术和改进的CDD模型的 优势,利用标架域中 小波系数之间具有的冗余性对缺损的信息进行弥补,对现有CDD模型进行改进。给出了 有效的split Bregman 仿真算法,并采用不同的图像进行了仿真。实验结果表明,新模型对大面积缺损修复及噪声 抑制都具有良好 的修复效果,即使在大量小波系数丢失的情况下,也能保持图像的边缘结构等几何特征,大 大的改善修复质量。     

A two-stage shearlet-based approach for the removal of random-valued impulse noise in images

In this paper, we introduce a novel two-stage denoising method for the removal of random-valued impulse noise (RVIN) in images. The first stage of our algorithm applies an impulse-noise detection routine that is a refinement of the HEIND algorithm and is very accurate in identifying the location of the noisy pixels. The second stage is an image inpainting routine that is designed to restore the missing information at those pixels that have been identified during the first stage. One of the novelties of our approach is that our inpainting routine takes advantage of the shearlet representation to efficiently recover the geometry of the original image. This method is particularly effective to eliminate jagged edges and other visual artifacts that frequently affect many RVIN denoising algorithms, especially at higher noise levels. We present extensive numerical demonstrations to show that our approach is very effective to remove random-valued impulse noise without any significant loss of fine-scale detail. Our algorithm compares very favourably against state-of-the-art methods in terms of both visual quality and quantitative measurements.     

基于样本和线性结构信息的大范围图像修复算法

基于纹理的图像修复算法对于修复破损区域比较大的图像效果较好,但该算法对于含有结构信息的图像修复效果很差.通过新的优先项的计算、平均值补偿及增加惩罚项提高传统的基于样本的图像修复算法的修复效果,结合图像中常出现的直线和曲线结构特征,提出了基于样本和结构信息的大范围图像修复算法.实验表明,该算法易于实现,修复结果能达到令人满意的效果,具有较高的实用价值.     

RESEARCH ON WEIGHTED PRIORITY OF EXEMPLAR-BASED IMAGE INPAINTING

The priority of the filled patch play a key role in the exemplar-based image inpainting, and it should be determined firstly to optimize the process of image inpainting. A modified image inpainting algorithm is proposed by weighted-priority based on the Criminisi algorithm. The improved algorithm demonstrates better relationship between the data term and the confidence term for the optimization of the priority than the classical Criminisi algorithm. By comparing the effect of the inpainted images with different structure, conclusion can be drawn that the optimal priority should be chosen properly for different images with different structures.     

Simultaneous structure and texture image inpainting

An algorithm for the simultaneous filling-in of texture and structure in regions of missing image information is presented in this paper. The basic idea is to first decompose the image into the sum of two functions with different basic characteristics, and then reconstruct each one of these functions separately with structure and texture filling-in algorithms. The first function used in the decomposition is of bounded variation, representing the underlying image structure, while the second function captures the texture and possible noise. The region of missing information in the bounded variation image is reconstructed using image inpainting algorithms, while the same region in the texture image is filled-in with texture synthesis techniques. The original image is then reconstructed adding back these two sub-images. The novel contribution of this paper is then in the combination of these three previously developed components, image decomposition with inpainting and texture synthesis, which permits the simultaneous use of filling-in algorithms that are suited for different image characteristics. Examples on real images show the advantages of this proposed approach.     

BINet: A binary inpainting network for deep patch-based image compression

Recent deep learning models outperform standard lossy image compression codecs. However, applying these models on a patch-by-patch basis requires that each image patch be encoded and decoded independently. The influence from adjacent patches is therefore lost, leading to block artefacts at low bitrates. We propose the Binary Inpainting Network (BINet), an autoencoder framework which incorporates binary inpainting to reinstate interdependencies between adjacent patches, for improved patch-based compression of still images. When decoding a patch, BINet additionally uses the binarised encodings from surrounding patches to guide its reconstruction. In contrast to sequential inpainting methods where patches are decoded based on previous reconstructions, BINet operates directly on the binary codes of surrounding patches without access to the original or reconstructed image data. Encoding and decoding can therefore be performed in parallel. We demonstrate that BINet improves the compression quality of a competitive deep image codec across a range of compression levels.     

基于几何模型的高质量虚拟视点绘制

该研究主要完成用kineet获得的“单路纹理+深度图像”来生成新的虚拟视点图像.针对深度图像的虚拟视点绘制技术(Depth Image Based Rendering,DIBR)生成的虚拟图像空洞问题,利用高斯混合模型分离前背景,背景空洞采用背景值填充,前景空洞采用改进的图像修复技术方法来填充.实验证明生成的虚拟视点图像具有较好的视觉效果.     

分层级联生成对抗网络用于手背静脉图像修复

为解决在识别过程中因手背静脉图像信息缺失而造成识别效率低下的问题,本文提出了分层级联生成对抗网络的手背静脉图像修复框架。该网络框架分别以级联与并行分层的方式进行修复操作,通过并行分层结构创新性的融合了不同静脉图像的特征信息;为有效地利用静脉图像的上下文信息对缺失的静脉图像信息进行预测与补全,在网络中创新性的引入了空洞卷积核与非局部注意力网络;为保证修复静脉图像质量与其真实图像的一致性,创新性的结合对抗损失与感知损失进行优化。实验结果表明,本文算法在视觉效果、峰值信噪比（Peak Signal to Noise Ratio,PSNR）和结构相似性（Structural Similarity Index,SSIM）等方面表现优于已有算法,并在两个公开的掌纹与指纹数据集上进行了有效的泛化验证。此外,修复图像相较于缺失图像在身份识别效率方面有了一定的提高。      

基于深度学习的数字图像修复算法最新进展

数字图像修复是一项利用计算机技术还原破损图像的缺失信息,从而实现自动修复破损图像的技术,其广泛应用于文物修复、图像去雾、电影特效生成等方面。近年来深度学习的发展为图像修复提供了新的思路,即将估计缺失信息的问题转为有条件的图像生成问题。基于深度学习的图像修复研究已成为底层计算机视觉问题的研究热点之一。本文对深度学习在数字图像修复领域的最新进展进行总结归纳,并详细阐述卷积模式和网络结构优化的研究进展,最后对未来的研究方向进行展望。      

Generative image inpainting with salient prior and relative total variation

Image inpainting is an important research direction of image processing. The generative adversarial network (GAN), which can reconstruct new reasonable content in the corrupted region, is the most interesting tool in current inpainting technologies. However, the previous deep methods generally need to be pre-added the binary mask representing the corruption location as the extra input. A novel inpainting algorithm which does not require additional external labels is proposed in this paper. The algorithm consists of two parts: corruption recognition module and content inpainting module, which can recognize and fill random corruption. In the recognizer, the salient object from the uncorrupted region is used as the prior for distinguishing corruption. In the inpainting module, a two-stage network is applied to reconstruct the image from coarse content to texture details. To avoid the misdetection in recognition which has a negative impact on the restoration in inpainting, we perform relative total variational filtering on the corrupted image, and use the salient map as the supervision of detail reconstruction. Qualitative and quantitative experiments on multiple datasets verify the effectiveness of our recognition module, the competitive advantage of our inpainting module, and the enlightening significance of our total algorithm in image inpainting.     

Unsupervised fisheye image correction through bidirectional loss with geometric prior

Neural network based methods for fisheye distortion correction are effective and increasingly popular, although training network require a high amount of labeled data. In this paper, we propose an unsupervised fisheye correction network to address the aforementioned issue. During the training process, the predicted parameters are employed to correct strong distortion that exists in the fisheye image and synthesize the corresponding distortion using the original distortion-free image. Thus, the network is constrained with bidirectional loss to obtain more accurate distortion parameters. We calculate the two losses at the image level as opposed to directly minimizing the difference between the predicted and ground truth of distortion parameters. Additionally, we leverage the geometric prior that the distortion distribution depends on the geometric regions of fisheye images and the straight line should be straight in the corrected images. The network focuses more on the geometric prior regions as opposed to equally perceiving the whole image without any attention mechanisms. To generate more appealing corrected results in visual appearance, we introduce a coarse-to-fine inpainting network to fill the hole regions caused by the irreversible mapping function using distortion parameters. Each module of the proposed network is differentiable, and thus the entire framework is completely end-to-end. When compared with the previous supervised methods, our method is more flexible and shows better practical applications for distortion rectification. The experiment results demonstrate that our proposed method outperforms state-of-the-art methods on the correction performance without any labeled distortion parameters.     

一种改进的BSCB图像修补算法

针对传统BSCB算法对颜色复杂度高或缺损区域较大的图像修补效果较差的问题,提出一种改进的BSCB图像修补算法.考虑传统算法中初始化、光滑算子和修补扩散等步骤中存在的缺陷,分别对其进行了改进.改进的BSCB算法进行图像修补时,采用随机初始化方法,引入平滑和梯度算子代替原拉普拉斯图像平滑算子,并采用加权平均算法选择所有邻点进行异向性扩散,从而得到最终修补结果.实验表明,新方法修补图像,特别是修补颜色复杂度高、缺损区域较大的图像具有较好的效果.