多尺度变换像素级医学图像融合：研究进展、应用和挑战

基于多尺度变换的像素级图像融合是计算机视觉领域的研究热点，广泛应用于医学图像处理等领域。本文对多尺度变换的像素级图像融合进行综述，阐述多尺度变换图像融合的基本原理和框架。在多尺度分解方面，以时间为序梳理了塔式分解、小波变换和多尺度几何分析方法的发展历程。在融合规则方面，围绕Piella框架和Zhang框架，讨论通用的像素级图像融合框架；在低频子带融合规则方面，总结基于像素、区域、模糊理论、稀疏表示和聚焦测度的5种融合规则；在高频子带融合规则方面，综述基于像素、边缘、区域、稀疏表示和神经网络的5种融合规则。总结12种跨模态医学图像融合方式，讨论该领域面临的主要挑战，并对未来的发展方向进行展望。本文系统梳理了多尺度变换像素级图像融合过程中的多尺度分解方法和融合规则，以及多尺度变换在医学图像融合中的应用，对多尺度变换像素级医学图像融合方法的研究具有积极的指导意义。

Research on pixel-level image fusion based on multi-scale transformation: progress application and challenges

Medical images can be classified as anatomical images and functional images. The high resolution in anatomical images can provide information about the anatomical structure and morphology of human body organs but cannot reflect the functional information of organs. There is some information about the metabolic functions of organs in functional images; however, it has low resolution and cannot display the anatomical details of organs or lesions. With the continuous development of medical imaging technology, how to use medical images comprehensively to avoid the shortcomings of different medical imaging modalities has become a complex problem. Image fusion has aimed to fuse value-added information from two images to one image in terms of the constraints of single sensor. Effective diagnose and treat diseases have been beneficial to acquire qualified fusion image of organs and tissues and lesions simultaneously via fusing positron emission tomography(PET) images and computed tomography images(CT). Research on pixel-level image fusion based on multi-scale transformation has been focused nowadays. A fusion image with rich edge information has been easy use under the circumstances of image registration. The detailed information for image fusion has been obtained via moderate image decomposition option and fusion strategical implementation. Pixel-level image fusion based on multi-scale transformation has been prior to medical image processing, video surveillance image processing, remote sensing image processing and other tasks. Pixel-level image fusion based on multi-scale transformation has been analyzed and summarized on the following five aspects:1) mechanism of multi-scale image fusion transformation;2) multiscale decomposition analysis;3) a consensus pixel-level fusion framework;4) subband fusion rules under high and low frequency; and 5) pixel-level image fusion based on multi-scale transformation application in medical image processing. First, the mechanism and framework of image fusion based on multi-scale transformation have been proposed. Next, the multi-temporal decomposition have been summarized via the pyramid decomposition evolvement, wavelet transform and multi-scale geometric analysis. Such as,the advantages of pyramid decomposition are simple implementation and fast calculation speed. The disadvantage of pyramid decomposition is that it has no directionality, sensitive to noise, has no stability during reconstruction, and there is redundancy between the layers of the pyramid. Wavelet transform does not have direction selectivity and shift invariance. The advantage of multi-scale geometric analysis is that it has multiple directions. The original image is decomposed into high-frequency subbands in different directions by nonsubsampled contourlet transform(NSCT), which can enhance the edge details of the image from many directions. Then, two consensus pixel-level fusion frameworks has been illustrated. The Zhang framework has demonstrated four analyses including activity level measurement, coefficient grouping method, coefficient combination method and consistency verification. The Piella framework has consisted of four ways on the aspects of matching measurement, activity measurement, decision module and synthesis module. Low frequency fused sub bands and high frequency fused sub-bands have been categorized. The low frequency fused sub band has been subdivided into five categories:pixel based, region based, fuzzy theory based, sparse representation based and focus metric based. Pixel-based fusion rules include average fusion rules, coefficient maximum fusion rules, etc., region-based fusion rules include regional variance maximum fusion rules, and local energy maximum fusion rules, etc., fuzzy theory fusion rules include fuzzy inference system fusion rules, etc. Fusion rules based on sparse representation include new sum of modified Laplacian, and extended sum of modified Laplacian and other fusion rules. Fusion rules based on focal measures include spatial frequency fusion rules, and local spatial frequencies fusion rules, etc. The high frequency fused sub band rules have been subdivided into five categories:pixel based, edge based, region based, sparse representation based and neural network based. Edge-based fusion rules include edge strength maximum fusion rules and guided filter-based fusion rules, etc., region-based fusion rules include average gradient fusion rules and improved average gradient fusion rules, etc., fusion rules based on sparse representation include multi-scale convolution sparse representation fusion rules and separable dictionary learning method fusion rules, etc. Fusion rules based on neural networks include parameter adaptive dual-channel pulse-coupled neural network, and adaptive dual-channel pulse-coupled neural network fusion rules, simplified pulse-coupled neural network and other fusion rules. At last,12 pixel-level image fusion based on multi-scale transformation have been summarized for multi-modal medical image fusions. Two-mode medical image fusion and three-mode medical image fusion have been identified. The two-mode medical image fusion has been subdivided into 11 categories. The CT/magnetic resonance image(MRI)/PET has been analyzed in three-mode medical image fusion. At last, the further 4 research areas of pixel-level image fusion based on multi-scale transformation have been summarized:1) accurate registration of image preprocessing issue;2) fusion of ultrasonic images with other multimodal medical images issue;3) integration of multi-algorithms for cascading decomposition; 4) multi-scale decomposition methods and fusion rules for pixel-level image fusion and the application of multi-scale transformation in medical image fusion have been summarized. This article systematically summarizes the multi-scale decomposition methods and fusion rules in the pixel-level image fusion process based on multi-scale transformation, as well as the application of multi-scale transformation in medical image fusion, and the research on the pixel-level medical image fusion method based on multi-scale transformation has a positive guiding significance.