彩色图像多尺度融合灰度化算法

为了使彩色图像灰度化后能够保留更多的原始特征,提出了一种新的基于多尺度图像融合的灰度化算法。将彩色图像分解为R、G、B三个通道图像,采用基于高斯-拉普拉斯金字塔的多尺度图像融合模型进行灰度化,并引入梯度域导向图像滤波（Gradient Domain Guided Image Filter,GGIF）来消除多尺度融合可能产生的伪影。将灰度化问题转换为保持三个通道单色图像特征的多尺度融合问题。实验结果表明,与其他灰度化算法相比,所提算法对边缘信息敏感,并能够在较亮或较暗区域检测出更多的图像细节特征。     

An intelligent fault diagnosis for machine maintenance using weighted soft-voting rule based multi-attention module with multi-scale information fusion

The ability of engineering systems to process multi-scale information is a crucial requirement in the development of an intelligent fault diagnosis model. This study develops a hybrid multi-scale convolutional neural network model coupled with multi-attention capability (HMS-MACNN) to solve both the inefficient and insufficient extrapolation problems of multi-scale models in fault diagnosis of a system operating in complex environments. The model's capabilities are demonstrated by its ability to capture the rich multi-scale characteristics of a gearbox including time and frequency multi-scale information. The capabilities of the Multi-Attention Module, which consists of an adaptive weighted rule and a novel weighted soft-voting rule, are respectively integrated to efficiently consider the contribution of each characteristic with different scales-to-faults at both feature- and decision-levels. The model is validated against experimental gearbox fault results and offers robustness and generalization capability with F1 value that is 27% higher than other existing multi-scale CNN-based models operating in a similar environment. Furthermore, the proposed model offers higher accuracy than other generic models and can accurately assign attention to features with different scales. This offers an excellent generalization performance due to its superior capability in capturing multi-scale information and in fusing advanced features following different fusion strategies by using Multi-Attention Module and the hybrid MS block compared to conventional CNN-based models.     

基于RCF的跨层融合特征的边缘检测

针对当前基于深度学习的边缘检测技术产生的边缘线条杂乱且模糊等问题,提出了一种基于RCF的端到端的跨层融合多尺度特征的边缘检测（CFF）模型。该模型使用RCF作为基线,在主干网络中加入CBAM,采用具有平移不变性的下采样技术,并且去除了主干网络中的部分下采样操作,以保留图像的细节信息,同时使用扩张卷积技术增大模型感受野。此外,采用跨层融合特征图的方式,使得高低层特征能够充分融合。为了平衡各阶段损失和融合损失之间的关系,以及避免出现多尺度特征融合之后低层细节过度丢失的现象,对每个损失添加了一个权重。在伯克利分割数据集（BSDS500）和PASCAL VOL Context数据集上进行了训练,在测试时使用图像金字塔技术提高边缘图像的质量。实验结果表明,CFF模型提取的轮廓比基线网络更加清晰,能够解决边缘模糊问题。在BSDS500基准上进行的评估表明,该模型将最佳数据集规模（ODS）和最佳图像比例（OIS）指标分别提高到0.818和0.839。     

Multi-view aggregation transformer for no-reference point cloud quality assessment

With the increasing maturity of 3D point cloud acquisition, storage, and transmission technologies, a large number of distorted point clouds without original reference exist in practical applications. Hence, it is necessary to design a no-reference point cloud quality assessment (PCQA) for point cloud systems. However, the existing no-reference PCQA metrics ignore the content differences and positional context among the projected images. For this, we propose a Multi-View Aggregation Transformer (MVAT) with two different fusion modules to extract the comprehensive feature representation of PCQA. Specifically, considering the content differences of different projected images, we first design a Content Fusion Module (CFM) to fuse multiple projected image features by adaptive weighting. Then, we design a Bidirectional Context Fusion Module (BCFM) to extract context features for reflecting the contextual relationship among projected images. Finally, we joint the above two fusion modules via Content-Position Fusion Module (CPFM) to fully mine the feature representation of point clouds. Experimental results show that our MVAT can achieve comparable or better performance than state-of-the-art metrics on three open point cloud datasets.     

Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with Gaussian and bilateral filters

In order to achieve perceptually better fusion of infrared (IR) and visible images than conventional pixel-level fusion algorithms based on multi-scale decomposition (MSD), we present a novel multi-scale fusion method based on a hybrid multi-scale decomposition (hybrid-MSD). The proposed hybrid-MSD transform decomposes the source images into multi-scale texture details and edge features by jointly using multi-scale Gaussian and bilateral filters. This transform enables to better capture important multi-scale IR spectral features and separate fine-scale texture details from large-scale edge features. As a result, we can use it to achieve better fusion result for human visual perception than those obtained from conventional multi-scale fusion methods, by injecting the multi-scale IR spectral features into the visible image, while preserving (or properly enhancing) important perceptual cues of the background scenery and details from the visible image. In the decomposed information fusion process, three different combination algorithms are adaptively used in accordance to different scale levels (i.e., the small-scale levels, the large-scale levels and the base level). A regularization parameter is introduced to control the relative amount of IR spectral information injected into the visible image in a soft manner, which can be adjusted further depending on user preferences. Moreover, by testing different settings of the parameter, we demonstrate that injecting a moderate amount of IR spectral information with this parameter can actually make the fused images visually better for some infrared and visible source images. Experimental results of both objective assessment and subjective evaluation by human observers also prove the superiority of the proposed method compared with conventional MSD-based fusion methods.     

基于多尺度边缘保持分解与PCNN的医学图像融合

在医学图像融合过程中,传统多尺度分析方法多采用线性滤波器,由于无法保留图像边缘特征导致分解阶段的强边缘处出现模糊,从而产生光晕。为提高融合图像的视觉感知效果,通过结合多尺度边缘保持分解方法与脉冲耦合神经网络（PCNN）,提出一种新的图像融合方法。对源图像进行加权最小二乘滤波分解得到图像的基础层和细节层,采用高斯滤波器对基础层进行二次分解得到低频层和边缘层,将分解过程中每级边缘层和细节层叠加构建高频层,并引入非下采样方向滤波器组进行方向分析。在此基础上,利用改进的空间频率以及区域能量激励PCNN融合高频层和低频层,通过逆变换得到最终的融合图像。实验结果表明,该方法能够突出医学图像的边缘轮廓并增强图像细节,可将更多的显著特征从源图像分离并转移到融合图像中。     

红外与可见光图像分解与融合方法研究

红外与可见光图像融合旨在生成一幅新的图像,能够对场景进行更全面的描述。本文提出一种图像多尺度混合信息分解方法,可有效提取代表可见光特征分量的纹理细节信息和代表红外特征分量的边缘信息。本文方法将边缘信息进行进一步分割以确定各分解子信息的融合权重,以有效地将多尺度红外光谱特征注入到可见光图像中,同时保留可见光图像中重要的场景细节信息。实验结果表明,本文方法能够有效提取图像中的红外目标,实现在融合图像中凸显红外目标的同时保留尽可能多的可见光纹理细节信息,无论是主观视觉还是客观评价指标都优于现有的图像融合方法。     

Relational-Convergent Transformer for image captioning

Image captioning describes the visual content of a given image by using natural language sentences, and plays a key role in the fusion and utilization of the image features. However, in the existing image captioning models, the decoder sometimes fails to efficiently capture the relationships between image features because of their lack of sequential dependencies. In this paper, we propose a Relational-Convergent Transformer (RCT) network to obtain complex intramodality representations in image captioning. In RCT, a Relational Fusion Module (RFM) is designed for capturing the local and global information of an image by a recursive fusion. Then, a Relational-Convergent Attention (RCA) is proposed, which is composed of a self-attention and a hierarchical fusion module for aggregating global relational information to extract a more comprehensive intramodal contextual representation. To validate the effectiveness of the proposed model, extensive experiments are conducted on the MSCOCO dataset. The experimental results show that the proposed method outperforms some of the state-of-the-art methods.     

Robust multi-modal medical image fusion via anisotropic heat diffusion guided low-rank structural analysis

This paper proposes a novel and robust multi-modal medical image fusion method, which is built upon a novel framework comprising multi-scale image decomposition based on anisotropic heat kernel design, scale-aware salient information extraction based on low-rank analysis, and scale-specific fusion rules. Our framework respects multi-scale structure features, while being robust to complex noise perturbation. First, anisotropic heat kernel is computed by constructing an image pyramid and embedding multi-level image properties into 2D manifolds in a divide-and-conquer way, consequently, multi-scale structure-preserving image decomposition can be accommodated. Second, to extract meaningfully scale-aware salient information, we conduct low-rank analysis over the image layer groups obtained in the first step, and employ the low-rank components to form the scale space of the salient features, wherein the underlying noise can be synchronously decoupled in a natural way. Third, to better fuse the complementary salient information extracted from multi-modal images, we design an S-shaped weighting function to fuse the large-scale layers, and employ the maximum selection principle to handle the small-scale layers. Moreover, we have conducted extensive experiments on MRI and PET/SPECT images. The comprehensive and quantitative comparisons with state-of-the-art methods demonstrate the informativeness, accuracy, robustness, and versatility of our novel approach.     

多尺度密集网络在红外和可见光图像融合应用

为了进一步提升红外和可见光图像的融合效果,提出了一种基于多尺度卷积算子和密集连接网络的图像融合模型.该模型首先使用多尺度卷积算子计算图像的直接多尺度特征,然后使用密集连接网络计算图像的间接多尺度特征.为了得到图像像素信息在不同尺度下的融合权重,通过叠加的方式将各个尺度密集连接网络的输出进行融合,并使用活动图方法计算两类图像的融合权重,最后根据权重计算结果得到融合图像,实验在THO数据集和CMA数据集获得较好的识别率.     

基于冗余小波变换与引导滤波的多聚焦图像融合

针对传统多尺度变换在多聚焦图像融合中存在的边缘晕圈问题,提出了一种基于冗余小波变换与引导滤波的多聚焦图像融合算法。首先,利用冗余小波变换对图像进行多尺度分解,将源图像分解为一个相似平面和一系列小波平面,该多尺度分解能够有效地提取源图像中的细节信息;然后,对相似平面和小波平面分别采用引导滤波的加权融合规则来构造加权映射,从而得到相似平面和小波平面的加权融合系数;最后,进行冗余小波逆变换,即可得到融合结果图。实验结果表明,与传统融合算法相比,所提算法能够更好地体现图像边缘的细节特征,取得了较好的融合效果。     

自适应融合特征的人群计数网络

针对人群计数方法中存在的尺度变化和多层级特征融合不佳的问题,基于U-Net的编码器-解码器网络结构,提出一种自适应特征融合网络,来进行精准的人群计数。提出自适应特征融合模块,根据解码器分支的需要,高效地聚合编码器分支提取的高层语义信息和底层的边缘信息;提出自适应上下文信息提取器,从不同感受野下提取多尺度的上下文信息并自适应加权融合,提高网络对于人头尺度变化的鲁棒性。在ShanghaiTech、UCF-CC-50和UCG-QNRF上的实验表明,与目前主流的人群计数算法相比,该算法具有更强的准确性和鲁棒性。     

DHIQA: Quality assessment of dehazed images based on attentive multi-scale feature fusion and rank learning

Haze is a ubiquitous atmospheric phenomenon that seriously influences the visibility of images. To this end, numerous image dehazing models have been proposed to improve the visual quality of hazy images. However, the quality assessment of dehazed images has fallen behind. Traditional IQA methods for dehazed images rely on complex handcrafted features that may not accurately evaluate the quality of dehazed images. In this paper, we present a novel no-reference image quality assessment network for dehazed images (called DHIQA). Firstly, we propose a multi-scale feature fusion network that considers typical dehazed distortion-related features, such as contrast, dark channel, edge, etc. Then, inspired by the contrast sensitivity function of the Human Visual System (HVS), we propose a contrast attentive module to enhance the effectiveness of the proposed IQA network. Moreover, in addition to the absolute quality score of dehazed images, their rankings are also important for quality assessment. Thus, we propose a new ranking loss, which takes both the absolute quality score and their rankings into consideration. Extensive experimental results on six publicly available dehazed IQA databases show that the proposed model achieves superior performance compared to current state-of-the-art metrics.     

多尺度变换像素级医学图像融合：研究进展、应用和挑战

基于多尺度变换的像素级图像融合是计算机视觉领域的研究热点,广泛应用于医学图像处理等领域。本文对多尺度变换的像素级图像融合进行综述,阐述多尺度变换图像融合的基本原理和框架。在多尺度分解方面,以时间为序梳理了塔式分解、小波变换和多尺度几何分析方法的发展历程。在融合规则方面,围绕Piella框架和Zhang框架,讨论通用的像素级图像融合框架;在低频子带融合规则方面,总结基于像素、区域、模糊理论、稀疏表示和聚焦测度的5种融合规则;在高频子带融合规则方面,综述基于像素、边缘、区域、稀疏表示和神经网络的5种融合规则。总结12种跨模态医学图像融合方式,讨论该领域面临的主要挑战,并对未来的发展方向进行展望。本文系统梳理了多尺度变换像素级图像融合过程中的多尺度分解方法和融合规则,以及多尺度变换在医学图像融合中的应用,对多尺度变换像素级医学图像融合方法的研究具有积极的指导意义。     

基于双向消息链路卷积网络的显著性物体检测

有效特征的提取和高效使用是显著性物体检测中极具挑战的任务之一。普通卷积神经网络很难兼顾提取有效特征和高效使用这些特征。本文提出双向消息链路卷积网络（bidirectional message link convolution network,BML-CNN）模型,提取和融合有效特征信息用于显著性物体检测。首先,利用注意力机制引导特征提取模块提取实体有效特征,并以渐进方式选择整合多层次之间的上下文信息。然后使用带有跳过连接结构的网络与带门控函数的消息传递链路组成的双向信息链路,将高层语义信息与浅层轮廓信息相融合。最后,使用多尺度融合策略,编码多层有效卷积特征,以生成最终显著图。实验表明,BML-CNN在不同指标下均获得最好的表现。     

基于互导滤波和显著性映射的红外可见光图像融合

为了解决传统多尺度红外可见光融合图像边缘模糊、对比度低和目标不显著的问题,提出一种基于互导滤波和显著性映射的红外可见光图像融合算法。由于互导滤波器能将图像一致结构和不一致结构分离并且具有尺度和保边意识,因此首先利用互导滤波器将原图像分解为具有冗余信息的结构层和不同尺度上具有互补信息的纹理层;其次根据过明或过暗区域更容易引起注意的视觉特点构造图像显著性映射函数对结构层和不同尺度的纹理层进行显著性映射;最后根据不同尺度的结构和纹理特性对图像进行融合重构。在两个数据集上的实验结果表明与传统的多尺度融合方法相比提出的方法在保持图像边缘、增强图像对比度、突出目标方面具有较好的主客观评价效果。     

结合语义辅助和边缘特征的显著对象检测

目的 现有的显著对象检测模型能够很好地定位显著对象,但是在获得完整均匀的对象和保留清晰边缘的任务上存在不足。为了得到整体均匀和边缘清晰的显著对象,本文提出了结合语义辅助和边缘特征的显著对象检测模型。方法 模型利用设计的语义辅助特征融合模块优化骨干网的侧向输出特征,每层特征通过语义辅助选择性融合相邻的低层特征,获得足够的结构信息并增强显著区域的特征强度,进而检测出整体均匀的显著对象。通过设计的边缘分支网络以及显著对象特征得到精确的边缘特征,将边缘特征融合到显著对象特征中,加强特征中显著对象边缘区域的可区分性,以便检测出清晰的边缘。同时,本文设计了一个双向多尺度模块来提取网络中的多尺度信息。结果 在4种常用的数据集ECSSD （extended complex scene saliency dataset）、DUT-O （Dalian University of Technology and OMRON Corporation）、HKU-IS和DUTS上与12种较流行的显著模型进行比较,本文模型的最大F值度量（max F-measure,MaxF）和平均绝对误差（mean absolution error,MAE）分别是0.940、0.795、0.929、0.870和0.041、0.057、0.034、0.043。从实验结果看,本文方法得到的显著图更接近真值图,在MaxF和MAE上取得最佳性能的次数多于其他12种方法。结论 本文提出的结合语义辅助和边缘特征的显著对象检测模型十分有效。语义辅助特征融合和边缘特征的引入使检测出的显著对象更为完整均匀,对象的边缘区分性也更强,多尺度特征提取进一步改善了显著对象的检测效果。     

红外与可见光图像渐进融合深度网络

目的 红外与可见光图像融合的目标是获得具有完整场景表达能力的高质量融合图像。由于深度特征具有良好的泛化性、鲁棒性和发展潜力,很多基于深度学习的融合方法被提出,在深度特征空间进行图像融合,并取得了良好的效果。此外,受传统基于多尺度分解的融合方法的启发,不同尺度的特征有利于保留源图像的更多信息。基于此,提出了一种新颖的渐进式红外与可见光图像融合框架(progressive fusion, ProFuse)。方法 该框架以U-Net为骨干提取多尺度特征,然后逐渐融合多尺度特征,既对包含全局信息的高层特征和包含更多细节的低层特征进行融合,也在原始尺寸特征(保持更多细节)和其他更小尺寸特征(保持语义信息)上进行融合,最终逐层重建融合图像。结果 实验在TNO(Toegepast Natuurwetenschappelijk Onderzoek)和INO(Institut National D’optique)数据集上与其他6种方法进行比较,在选择的6项客观指标上,本文方法在互信息(mutual Information, MI)上相比FusionGAN(generative adversarial ...     

Contourlet变换系数加权的医学图像融合

目的 由于获取医学图像的原理和设备不同,不同模式所成图像的质量、空间与时间特性都有较大差别,并且不同模式成像提供了不互相覆盖的互补信息,临床上通常需要对几幅图像进行综合分析来获取信息。方法 为了提高对多源图像融合信息的理解能力,结合Contourlet变换在多尺度和多方向分析方法的优势,将Contourlet变换应用于医学图像融合中。首先将源图像经过Contourlet变换分解获得不同尺度多个方向下的分解系数。其次通过对Contourlet变换后的系数进行分析来确定融合规则。融合规则主要体现在Contourlet变换后图像中的低频子带系数与高频子带系数的优化处理中。针对低频子带主要反映图像细节的特点,对低频子带系数采用区域方差加权融合规则;针对高频子带系数包含图像中有用边缘细节信息的特点,对高频子带系数采用基于主图像的条件加权融合规则。最后经过Contourlet变换重构获得最终融合图像。结果 分别进行了基于Contourlet变换的不同融合规则实验对比分析和不同融合方法实验对比分析。通过主观视觉效果及客观评价指标进行评价,并与传统融合算法进行比较,该算法能够克服融合图像在边缘及轮廓部分变得相对模糊的问题,并能有效地融合多源医学图像信息。结论 提出了一种基于Contourlet变换的区域方差加权和条件加权融合算法。通过对CT与MRI脑部医学图像的仿真实验表明,该算法可以增加多模态医学图像互补信息,并能较好地提高医学图像融合的清晰度。     

基于动态注意力机制和多模态循环融合的帕金森氏症识别

PD （Parkinson’s disease） 的运动障碍会累及口、咽、腭肌以及面部肌肉,引起声带震颤和面部运动迟缓,为利用声纹和面部特征识别PD患者提供了可能。为了有效利用以上两种特征以提高PD 识别率,提出了基于多尺度特征与动态注意力机制的多模态循环融合模型对患者进行识别检测。首先,设计了多尺度特征提取网络,将高、低层级特征的语义信息融合以得到完整的特征信息;其次,在多尺度特征融合过程中为了充分考虑模态间的相关性和互补性,提出了以不同模态信息互为辅助条件生成注意力特征图的动态注意力机制算法,降低特征融合时信息的冗余;最后设计了多模态循环融合模型,通过计算循环矩阵的每个行向量与特征向量间的哈达玛积得到更有效的融合特征,提高了模型性能。在自建数据集上进行的多组实验结果表明,提出的方法识别准确率高达96.24%,优于当前流行的单模态和多模态识别算法,可以有效区分PD患者和HP （healthy people）,为高效识别PD患者奠定了基础。