自学习规则下的多聚焦图像融合

目的 基于深度学习的多聚焦图像融合方法主要是利用卷积神经网络（convolutional neural network,CNN）将像素分类为聚焦与散焦。监督学习过程常使用人造数据集,标签数据的精确度直接影响了分类精确度,从而影响后续手工设计融合规则的准确度与全聚焦图像的融合效果。为了使融合网络可以自适应地调整融合规则,提出了一种基于自学习融合规则的多聚焦图像融合算法。方法 采用自编码网络架构,提取特征,同时学习融合规则和重构规则,以实现无监督的端到端融合网络;将多聚焦图像的初始决策图作为先验输入,学习图像丰富的细节信息;在损失函数中加入局部策略,包含结构相似度（structural similarity index measure,SSIM）和均方误差（mean squared error,MSE）,以确保更加准确地还原图像。结果 在Lytro等公开数据集上从主观和客观角度对本文模型进行评价,以验证融合算法设计的合理性。从主观评价来看,模型不仅可以较好地融合聚焦区域,有效避免融合图像中出现伪影,而且能够保留足够的细节信息,视觉效果自然清晰;从客观评价来看,通过将模型融合的图像与其他主流多聚焦图像融合算法的融合图像进行量化比较,在熵、Q_w、相关系数和视觉信息保真度上的平均精度均为最优,分别为7.457 4,0.917 7,0.978 8和0.890 8。结论 提出了一种用于多聚焦图像的融合算法,不仅能够对融合规则进行自学习、调整,并且融合图像效果可与现有方法媲美,有助于进一步理解基于深度学习的多聚焦图像融合机制。     

基于多尺度注意力半监督学习的老照片划痕修复

老照片由于长时间的磨损或保存不当,会出现照片的划痕损伤。随着深度学习在图像重建中的应用,基于深度学习方法能够在纹理修复的基础上获取图像的语义信息并预测语义内容,使老照片修复的整体效果更加符合客观事实,但利用深度学习进行老照片划痕修复缺乏学习所需数据集。提出一种基于半监督学习的老照片划痕自动修复的方法,创建划痕合成数据集SynOld用于网络训练,同时搜集真实的划痕老照片用于训练和测试,将合成数据集和真实老照片加入网络学习,两者共享网络参数,并通过鉴别器来区分网络生成图像与真实图像。对于合成数据集有监督的分支采用均方差损失、感知损失和对抗损失约束训练,对于真实老照片无监督的分支采用总变差损失控制训练。实验结果表明,相比于多尺度特征注意力网络的监督学习方法,该方法在合成数据集SynOld和真实老照片上都具有较好的修复效果。     

一种结合类激活映射的半监督图像分类方法

半监督学习要求无标记数据集远大于标记数据集,然而无标记数据集中包含的复杂无关信息又会对模型训练造成负面影响.针对此问题,本文提出了一种基于增强的均值教师模型的半监督图像分类方法.首先根据类激活映射的工作机制,构建一个具有类激活映射功能的网络;然后将无标记数据集输入结合类激活映射的目标初定位网络,得到目标初定位图;最后将标记图像和目标初定位图像组成训练数据集,训练得到半监督图像分类器.本文设置了标记数据占比、骨干网络、数据集的对比实验,结果表明,本文算法在Top1和Top5上的表现优于现有算法,说明了本文算法的可行性和有效性.     

多标记学习自编码网络无监督维数约简

多标记学习是针对一个实例同时与一组标签相关联而提出的一种机器学习框架,是该领域研究热点之一,降维是多标记学习一个重要且具有挑战性的工作。针对有监督的多标记维数约简方法,提出一种无监督自编码网络的多标记降维方法。首先,通过构建自编码神经网络,对输入数据进行编码和解码输出;然后,引入稀疏约束计算总体成本,使用梯度下降法进行迭代求解;最后,通过深度学习训练获得自编码网络学习模型,提取数据特征实现维数约简。实验中使用多标记算法ML-kNN做分类器,在6个公开数据集上与其他4种方法对比。实验结果表明,该方法能够在不使用标记的情况下有效提取特征,降低多标记数据维度,稳定提高多标记学习性能。     

多层自适应聚合的自监督小样本图像分类

目的 在图像分类领域,小样本学习旨在利用从大规模数据集中训练到的知识来处理仅包含少量有标记训练样本的下游分类任务。通常情况下,下游任务只涉及新类样本,由于元训练阶段会构造大量任务随机抽取训练集中不同类别的样本且训练集与测试集类别间存在领域间隙,因此模型训练周期长且可能对训练集过拟合,以致元知识无法迁移到测试集,进而导致模型泛化性差。针对以上问题,提出一种多层自适应聚合的自监督小样本图像分类模型。方法 首先使用分组卷积对残差块进行改进,减少神经网络参数量,降低训练难度,缩短训练时间;然后采用多层自适应聚合的方法改进骨干网络,对网络各层语义信息加以提炼聚合,自适应分配各层权重,将聚合后的特征图作为后续分类的依据;最后加入自监督对比学习结合有监督学习挖掘样本自身潜在的信息,从而提升样本特征表达能力。结果 在mini-ImageNet数据集和CUB(Caltech-UCSD birds-200-2011)数据集上与当前主流模型进行分类效果对比实验,与baseline相比,所提模型的准确率在mini-ImageNet数据集的5-way 1-shot与5-way 5-shot实验上分别提升了6.3...     

小样本下基于Wasserstein距离的半监督学习算法

近年来,基于大规模标记数据集的深度神经网络模型在图像领域展现出优秀的性能,但是大量标记数据昂贵且难以收集.为了更好地利用无标记数据,提出了一种半监督学习方法Wasserstein consistency training(WCT),通过引入Jensen-Shannon散度来模拟协同训练并组织大量未标记数据来提高协同训练...     

基于联合卷积自编码网络的多聚焦图像融合方法

受相机景深的限制,单次成像无法对不同景深的内容全部清晰成像.多聚焦图像融合技术可以将不同聚焦层次的图像融合为一幅全聚焦的图像,其中如何得到准确的聚焦映射是多聚焦图像融合中的关键问题.对此,利用卷积神经网络强大的特征提取能力,设计具有公共分支和私有分支的联合卷积自编码网络以学习多源图像的特征,公共分支学习多幅图像之间的公共特征,每幅图像的私有分支学习该图像区别于其他图像的私有特征.基于私有特征计算图像的活动测度,得到图像聚焦区域映射,据此设计融合规则以融合两幅多聚焦图像,最终得到全聚焦的融合图像.在公开数据集上的对比实验结果显示:主观评测上,所提出的方法能够较好地融合聚焦区域,视觉效果自然清晰;客观指标上,该方法在多个评价指标上优于对比方法.     

结合多注意力机制的自监督目标跟踪

为解决现有目标跟踪数据集不足及手工标注数据成本过大的问题,提出结合判别式相关滤波及多注意力机制的自监督目标跟踪方法.训练集选用原始未标记的视频图像,使用子空间注意力机制及通道注意力机制针对不同的输入目标对特征进行自适应调整,构建相关滤波输出响应图进行跟踪定位;通过前向跟踪和后向跟踪两个跟踪过程,以自监督的训练方式用最终响应结果与初始标签构建循环一致性损失.在O T B50和O T B100两个公开数据集的实验结果表明了所提方法的实时性和有效性.     

基于多特征融合的无监督真值发现方法

真值发现是数据集成领域具有挑战性的研究热点之一。传统的方法利用数据源与观测值之间的交互关系推断真值,缺乏足够的特征信息;基于深度学习的方法可以有效地进行特征抽取,但其性能依赖于大量手工标注,而在实际应用中很难获取到大量高质量的真值标签。为克服以上问题,本文提出一种基于多特征融合的无监督真值发现方法（Unsupervised truth discovery method based on multi-feature fusion, MFOTD）。首先,利用集成学习无监督标注“真值”标签;然后,分别使用预训练模型 Bert和独热编码获取观测值的语义特征和交互特征;最后,融合观测值多种特征并使用其“真值”标签构建初始训练集,通过自训练方式训练真值预测模型。在两个真实数据集上的实验结果表明,与已有方法相比,本文所提出的方法具有更高的真值发现准确性。     

基于自监督增强特征的直推式零样本图像分类

图像的视觉特征对实现零样本图像分类有至关重要的作用.尽管目前VGG、GoogLeNet和ResNet等网络提取的深度特征在图像分类领域获得了广泛的应用,但其在零样本图像分类问题上的表现并不理想,仍旧存在较大的提升空间.此外,由于零样本学习场景下训练集与测试集不相交的设定,导致分类网络不可避免地存在领域偏移问题.为此,提出一种基于自监督增强特征的直推式零样本图像分类框架.首先,通过辅助任务构造伪标签,利用自监督学习获得图像的自监督特征并将其与无监督深度特征进行特征融合;然后,将融合特征嵌入语义空间中进行零样本图像分类,并获得未见类的初始预测标签;最后,利用未见类特征和预测标签迭代地优化视觉-语义映射.所提出框架组件可选择,框架组件自监督网络、主干网络和降维网络分别选用CFN、VGG16和PCA构成网络.在CUB、SUN和AwA2数据集上的实验结果表明,所提出网络能够增强特征的判别能力,在零样本图像分类问题上表现良好.     

Ensemble of CNN for multi-focus image fusion

The Convolutional Neural Networks (CNNs) based multi-focus image fusion methods have recently attracted enormous attention. They greatly enhanced the constructed decision map compared with the previous state of the art methods that have been done in the spatial and transform domains. Nevertheless, these methods have not reached to the satisfactory initial decision map, and they need to undergo vast post-processing algorithms to achieve a satisfactory decision map. In this paper, a novel CNNs based method with the help of the ensemble learning is proposed. It is very reasonable to use various models and datasets rather than just one. The ensemble learning based methods intend to pursue increasing diversity among the models and datasets in order to decrease the problem of the overfitting on the training dataset. It is obvious that the results of an ensemble of CNNs are better than just one single CNNs. Also, the proposed method introduces a new simple type of multi-focus images dataset. It simply changes the arranging of the patches of the multi-focus datasets, which is very useful for obtaining the better accuracy. With this new type arrangement of datasets, the three different datasets including the original and the Gradient in directions of vertical and horizontal patches are generated from the COCO dataset. Therefore, the proposed method introduces a new network that three CNNs models which have been trained on three different created datasets to construct the initial segmented decision map. These ideas greatly improve the initial segmented decision map of the proposed method which is similar, or even better than, the other final decision map of CNNs based methods obtained after applying many post-processing algorithms. Many real multi-focus test images are used in our experiments, and the results are compared with quantitative and qualitative criteria. The obtained experimental results indicate that the proposed CNNs based network is more accurate and have the better decision map without post-processing algorithms than the other existing state of the art multi-focus fusion methods which used many post-processing algorithms.     

A multi-focus image fusion method based on attention mechanism and supervised learning

Multi-focus image fusion is always a difficult problem in digital image processing. To achieve efficient integration, we propose a new end-to-end network. This network uses the residual atrous spatial pyramid pooling module to extract multi-level features from the space of different scales and share parameters to ensure the consistency and correspondence of features. We also introduced a disparities attention module for the network which allows for information retention. These two parts can make our method overcome the difficulties of target edge artifacts, small range blur, poor detail capture, and so on. In addition, in order to improve the semantic ambiguity easily caused by unsupervised learning, we also proposed a new multi-focus image fusion dataset with groundtruth for supervised learning. We performed sufficient experiments, and the results show that the network can quickly capture the corresponding features of multi-focus images, and improve the fusion performance with less computation and lower storage cost. Compared with the existing nine fusion methods, our network is superior to other methods in subjective visual evaluation and objective evaluation, reaching a higher level.

     

Multi-focus image fusion using dictionary-based sparse representation

Multi-focus image fusion has emerged as a major topic in image processing to generate all-focus images with increased depth-of-field from multi-focus photographs. Different approaches have been used in spatial or transform domain for this purpose. But most of them are subject to one or more of image fusion quality degradations such as blocking artifacts, ringing effects, artificial edges, halo artifacts, contrast decrease, sharpness reduction, and misalignment of decision map with object boundaries. In this paper we present a novel multi-focus image fusion method in spatial domain that utilizes a dictionary which is learned from local patches of source images. Sparse representation of relative sharpness measure over this trained dictionary are pooled together to get the corresponding pooled features. Correlation of the pooled features with sparse representations of input images produces a pixel level score for decision map of fusion. Final regularized decision map is obtained using Markov Random Field (MRF) optimization. We also gathered a new color multi-focus image dataset which has more variety than traditional multi-focus image sets. Experimental results demonstrate that our proposed method outperforms existing state-of-the-art methods, in terms of visual and quantitative evaluations.     

MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion

Multi-focus image fusion is an enhancement method to generate full-clear images, which can address the depth-of-field limitation in imaging of optical lenses. Most existing methods generate the decision map to realize multi-focus image fusion, which usually lead to detail loss due to misclassification, especially near the boundary line of the focused and defocused regions. To overcome this challenge, this paper presents a new generative adversarial network with adaptive and gradient joint constraints to fuse multi-focus images. In our model, an adaptive decision block is introduced to determine whether source pixels are focused or not based on the difference of repeated blur. Under its guidance, a specifically designed content loss can dynamically guide the optimization trend, that is, force the generator to produce a fused result of the same distribution as the focused source images. To further enhance the texture details, we establish an adversarial game so that the gradient map of the fused result approximates the joint gradient map constructed based on the source images. Our model is unsupervised without requiring ground-truth fused images for training. In addition, we release a new dataset containing 120 high-quality multi-focus image pairs for benchmark evaluation. Experimental results demonstrate the superiority of our method over the state-of-the-art in terms of both subjective visual effect and quantitative metrics. Moreover, our method is about one order of magnitude faster compared with the state-of-the-art.     

深度学习时代图像融合技术进展

为提升真实场景视觉信号的采集质量,往往需要通过多种融合方式获取相应的图像,例如,多聚焦、多曝光、多光谱和多模态等。针对视觉信号采集的以上特性,图像融合技术旨在利用同一场景不同视觉信号的优势,生成单图像信息描述,提升视觉低、中、高级任务的性能。目前,依托端对端学习强大的特征提取、表征及重构能力,深度学习已成为图像融合研究的主流技术。与传统图像融合技术相比,基于深度学习的图像融合模型性能显著提高。随着深度学习研究的深入,一些新颖的理论和方法也促进了图像融合技术的发展,如生成对抗网络、注意力机制、视觉Transformer和感知损失函数等。为厘清基于深度学习技术的图像融合研究进展,本文首先介绍了图像融合问题建模,并从传统方法视角逐渐向深度学习视角过渡。具体地,从数据集生成、神经网络构造、损失函数设计、模型优化和性能评估等方面总结了基于深度学习的图像融合研究现状。此外,还讨论了选择性图像融合这类衍生问题建模(如基于高分辨率纹理图融合的深度图增强),回顾了一些基于图像融合实现其他视觉任务的代表性工作。最后,根据现有技术的缺陷,提出目前图像融合技术存在的挑战,并对未来发展趋势给出了展望。     

基于引导滤波和差分图像的多聚焦图像融合方法

针对传统的多聚焦图像的空间域融合容易出现边缘模糊的问题,提出了一种基于引导滤波（GF）和差分图像的多聚焦图像融合方法。首先,将源图像进行不同水平的GF,并对滤波后图像进行差分,从而获得聚焦特征图像;随后,利用聚焦特征图像的梯度能量（EOG）信息获得初始决策图,对初始决策图进行空间一致性检查以及形态学操作以消除因EOG相近而造成的噪点;然后,对初始决策图进行GF以得到优化后决策图,从而避免融合后的图像存在边缘骤变的问题;最后,基于优化后决策图对源图像进行加权融合,以得到融合图像。选取3组经典的多聚焦图像作为实验图像,将所提方法与其他9种多聚焦图像融合方法得到的结果进行比较。主观视觉效果显示,所提方法能更好地将多聚焦图像的细节信息保存下来,另外,经该方法处理后的图像的4项客观评价指标均显著优于对比方法。结果表明,所提方法能够获得高质量的融合图像,较好地保留原始图像信息,有效解决传统多聚焦图像融合出现的边缘模糊问题。     

Multi-scale convolutional neural network for multi-focus image fusion

In this study, we present new deep learning (DL) method for fusing multi-focus images. Current multi-focus image fusion (MFIF) approaches based on DL methods mainly treat MFIF as a classification task. These methods use a convolutional neural network (CNN) as a classifier to identify pixels as focused or defocused pixels. However, due to unavailability of labeled data to train networks, existing DL-based supervised models for MFIF add Gaussian blur in focused images to produce training data. DL-based unsupervised models are also too simple and only applicable to perform fusion tasks other than MFIF. To address the above issues, we proposed a new MFIF method, which aims to learn feature extraction, fusion and reconstruction components together to produce a complete unsupervised end-to-end trainable deep CNN. To enhance the feature extraction capability of CNN, we introduce a Siamese multi-scale feature extraction module to achieve a promising performance. In our proposed network we applied multiscale convolutions along with skip connections to extract more useful common features from a multi-focus image pair. Instead of using basic loss functions to train the CNN, our model utilizes structure similarity (SSIM) measure as a training loss function. Moreover, the fused images are reconstructed in a multiscale manner to guarantee more accurate restoration of images. Our proposed model can process images with variable size during testing and validation. Experimental results on various test images validate that our proposed method yields better quality fused images that are superior to the fused images generated by compared state-of-the-art image fusion methods.     

基于LDA和卷积神经网络的半监督图像标注方法

随着智能设备的不断出现,图像数量急速增加,但是很多图像因为没有被标注所以未被充分利用.为了能够使该问题得到较好解决,提出了基于LDA和卷积神经网络的半监督图像标注方法.首先把图像训练集中的所有文字信息放入LDA中,生成图像的文字标注词;然后使用卷积神经网络获得图像的高层视觉特征,同时用加入注意力机制和修改损失函数的方法...     

Multi-focus image fusion with deep residual learning and focus property detection

Multi-focus image fusion methods can be mainly divided into two categories: transform domain methods and spatial domain methods. Recent emerged deep learning (DL)-based methods actually satisfy this taxonomy as well. In this paper, we propose a novel DL-based multi-focus image fusion method that can combine the complementary advantages of transform domain methods and spatial domain methods. Specifically, a residual architecture that includes a multi-scale feature extraction module and a dual-attention module is designed as the basic unit of a deep convolutional network, which is firstly used to obtain an initial fused image from the source images. Then, the trained network is further employed to extract features from the initial fused image and the source images for a similarity comparison, aiming to detect the focus property of each source pixel. The final fused image is obtained by selecting corresponding pixels from the source images and the initial fused image according to the focus property map. Experimental results show that the proposed method can effectively preserve the original focus information from the source images and prevent visual artifacts around the boundary regions, leading to more competitive qualitative and quantitative performance when compared with the state-of-the-art fusion methods.