Infrared and visible image fusion scheme based on NSCT and low-level visual features

Multi-scale transform (MST) is an efficient tool for image fusion. Recently, many fusion methods have been developed based on different MSTs, and they have shown potential application in many fields. In this paper, we propose an effective infrared and visible image fusion scheme in nonsubsampled contourlet transform (NSCT) domain, in which the NSCT is firstly employed to decompose each of the source images into a series of high frequency subbands and one low frequency subband. To improve the fusion performance we designed two new activity measures for fusion of the lowpass subbands and the highpass subbands. These measures are developed based on the fact that the human visual system (HVS) percept the image quality mainly according to its some low-level features. Then, the selection principles of different subbands are presented based on the corresponding activity measures. Finally, the merged subbands are constructed according to the selection principles, and the final fused image is produced by applying the inverse NSCT on these merged subbands. Experimental results demonstrate the effectiveness and superiority of the proposed method over the state-of-the-art fusion methods in terms of both visual effect and objective evaluation results.     

Visual attention guided image fusion with sparse representation

Image fusion techniques aim at transferring useful information from the input source images to the fused image. The common assumption for most fusion approaches is that the useful information is defined by local features such as contrast, variance, and gradient. However, there is no consideration of global visual attention of the whole source images which indicates the “interesting” information of the source images. In this paper, we firstly review the patch-based image fusion methods which attract the attention and interest of many researchers. Then, a visual attention guided patch-based image fusion method is proposed. The visual attention maps of the source images are calculated from the sparse represent coefficients of the source images. Then, the sparse coefficients are fused with the guidance of visual attention maps in order to emphasize the global “interesting” objects in the source images. Finally, the fused image is reconstructed from the fused sparse coefficients. The new fusion strategy ensures that the objects being “interesting” for our visual system are preserved in the fused image. The proposed approach is tested on infrared and visual, medical, and multi-focus images. The results compared with those of traditional methods show obvious improvement in objective and subjective quality measurements.     

A fusion method for visible and infrared images based on contrast pyramid with teaching learning based optimization

This paper proposes a novel image fusion scheme based on contrast pyramid (CP) with teaching learning based optimization (TLBO) for visible and infrared images under different spectrum of complicated scene. Firstly, CP decomposition is employed into every level of each original image. Then, we introduce TLBO to optimizing fusion coefficients, which will be changed under teaching phase and learner phase of TLBO, so that the weighted coefficients can be automatically adjusted according to fitness function, namely the evaluation standards of image quality. At last, obtain fusion results by the inverse transformation of CP. Compared with existing methods, experimental results show that our method is effective and the fused images are more suitable for further human visual or machine perception.     

基于结构相似度的图像融合质量评价

在分析现有图像融合质量评价方法特点的基础上，提出了新型的、基于结构相似度的图像融合质量评价方法.针对不同情况，分别采用平均结构相似度、加权平均结构相似度、结构信息与交互信息量之乘积作为图像融合质量客观评价标准.该方法充分考虑了图像的结构信息和人类视觉系统的特性，可以为不同场合下选择不同的算法提供依据.对不同融合算法的质量评价结果表明,该方法是一种有效的图像融合质量评价方法.     

Fusion method of infrared and visible images based on neighborhood characteristic and regionalization in NSCT domain

On fusing infrared and visible image, the traditional fusion method cannot get the better image quality. Based on neighborhood characteristic and regionalization in NSCT (Nonsubsampled Contourlet Transform) domain, the fusion algorithm was proposed. Firstly, NSCT was adopted to decompose infrared and visible images at different scales and directions for the low and high frequency coefficients, the low frequency coefficients which were fused with improving regional weighted fusion method based on neighborhood energy, and the high-frequency coefficients were fused with multi-judgment rule based on neighborhood characteristic regional process. Finally, the coefficients were reconstructed to obtain the fused image. The experimental results show that, compared with the other three related methods, the proposed method can get the biggest value of IE (information entropy), MI(VI,F) (mutual information from visible image), MI(VI,F) (mutual information from infrared image), MI (sum of mutual information), and Q^AB/F (edge retention). The proposed method can leave enough information in the original images and its details, and the fused images have better visual effects.     

Two-scale image fusion of visible and infrared images using saliency detection

Military, navigation and concealed weapon detection need different imaging modalities such as visible and infrared to monitor a targeted scene. These modalities provide complementary information. For better situation awareness, complementary information of these images has to be integrated into a single image. Image fusion is the process of integrating complementary source information into a composite image. In this paper, we propose a new image fusion method based on saliency detection and two-scale image decomposition. This method is beneficial because the visual saliency extraction process introduced in this paper can highlight the saliency information of source images very well. A new weight map construction process based on visual saliency is proposed. This process is able to integrate the visually significant information of source images into the fused image. In contrast to most of the multi-scale image fusion techniques, proposed technique uses only two-scale image decomposition. So it is fast and efficient. Our method is tested on several image pairs and is evaluated qualitatively by visual inspection and quantitatively using objective fusion metrics. Outcomes of the proposed method are compared with the state-of-art multi-scale fusion techniques. Results reveal that the proposed method performance is comparable or superior to the existing methods.     

Infrared and visible image fusion based on visual saliency map and weighted least square optimization

The goal of infrared (IR) and visible image fusion is to produce a more informative image for human observation or some other computer vision tasks. In this paper, we propose a novel multi-scale fusion method based on visual saliency map (VSM) and weighted least square (WLS) optimization, aiming to overcome some common deficiencies of conventional methods. Firstly, we introduce a multi-scale decomposition (MSD) using the rolling guidance filter (RGF) and Gaussian filter to decompose input images into base and detail layers. Compared with conventional MSDs, this MSD can achieve the unique property of preserving the information of specific scales and reducing halos near edges. Secondly, we argue that the base layers obtained by most MSDs would contain a certain amount of residual low-frequency information, which is important for controlling the contrast and overall visual appearance of the fused image, and the conventional “averaging” fusion scheme is unable to achieve desired effects. To address this problem, an improved VSM-based technique is proposed to fuse the base layers. Lastly, a novel WLS optimization scheme is proposed to fuse the detail layers. This optimization aims to transfer more visual details and less irrelevant IR details or noise into the fused image. As a result, the fused image details would appear more naturally and be suitable for human visual perception. Experimental results demonstrate that our method can achieve a superior performance compared with other fusion methods in both subjective and objective assessments.     

A novel image fusion algorithm based on nonsubsampled shearlet transform

To overcome the shortcoming of traditional image fusion method based on multi-scale transform, a novel adaptive image fusion algorithm based on nonsubsampled shearlet transform (NSST) is proposed. Firstly, the NSST is utilized to decompose the source images on various scales and in different directions, and the low frequency sub-band and bandpass sub-band coefficients are obtained. Secondly, for the low frequency sub-band coefficients, the singular value decomposition method in the gradient domain is used to estimate the local structure information of image, and an adaptive ‘weighted averaging’ fusion rule based on the sigmoid function and the extracted features is presented. To improve the quality of fused image, a novel sum-modified-Laplacian (NSML), which can extract more useful information from source images, is employed as the measurement to select bandpass sub-band coefficients. Finally, the fused image is obtained by performing the inverse NSST on the combined coefficients. The proposed fusion method is verified on several sets of multi-source images, and the experimental results show that the proposed approach can significantly outperform the conventional image fusion methods in terms of both objective evaluation criteria and visual quality.     

Fusion of visible and infrared images using global entropy and gradient constrained regularization

Infrared and visible image fusion has been an important and popular topic in imaging science. Dual-band image fusion aims to extract both target regions in infrared image and abundant detail information in visible image into fused result, preserving even enhancing the information that inherits from source images. In our study, we propose an optimization-based fusion method by combining global entropy and gradient constrained regularization. We design a cost function by taking the advantages of global maximum entropy as the first term, together with gradient constraint as the regularized term. In this cost function, global maximum entropy could make the fused result inherit as more information as possible from sources. And using gradient constraint, the fused result would have clear details and edges with noise suppression. The fusion is achieved based on the minimization of the cost function by adding weight value matrix. Experimental results indicate that the proposed method performs well and has obvious superiorities over other typical algorithms in both subjective visual performance and objective criteria.     

基于人眼视觉系统的假彩色融合图像质量的评价方法

随着图像融合技术的发展,各种融合算法层出不穷,而很多情况下最终的融合图像是由人眼观察的,因此基于人眼视觉系统的图像融合质量评价显得尤为重要.为了能够模拟人眼对于融合图像的感知,得到融合后图像质量的客观评价,本文提出了一种基于色差理论的假彩色融合图像质量的评价方法.首先将源图像和融合图像转化到CIE L*a*b*均匀色空间,在频域对图像进行对比度敏感函数滤波,通过计算滤波后融合图像的色差判断图像的细节信息,在一定程度上色差越大信息越丰富;通过计算融合图像与源图像的色差判断融合图像与源图像的相关性,相关性越高,融合算法越好.通过融合图像的色差大小以及与源图像的相关性两个参量,得出融合算法的优劣.实验表明,与其他评价方法相比,本文提出的评价方法与人眼观察的结果较为一致.     

高准确度光学系统中材料非均匀性对成像质量的影响

利用Zernike多项式对用Zygo干涉仪测得的离散材料折射率数据进行了拟合,再使用光线光学的方法评价了系统的成像质量. 由于材料折射率分布的无规则性,在对包含非均匀介质的实际光学系统的模拟仿真和优化时,需要考虑选取材料不同部位加工成的透镜会对系统成像质量有不同的影响,而且加工好的透镜在装配过程中,绕着光轴旋转不同的角度同样会影响成像质量. 通过计算机模拟的方法预先选取材料的最佳部位以及找到最好的装配位置,从而提高了光学系统的性能.     

Infrared image enhancement with learned features

Due to the variation of imaging environment and limitations of infrared imaging sensors, infrared images usually have some drawbacks: low contrast, few details and indistinct edges. Hence, to promote the applications of infrared imaging technology, it is essential to improve the qualities of infrared images. To enhance image details and edges adaptively, we propose an infrared image enhancement method under the proposed image enhancement scheme. On the one hand, on the assumption of high-quality image taking more evident structure singularities than low-quality images, we propose an image enhancement scheme that depends on the extractions of structure features. On the other hand, different from the current image enhancement algorithms based on deep learning networks that try to train and build the end-to-end mappings on improving image quality, we analyze the significance of first layer in Stacked Sparse Denoising Auto-encoder and propose a novel feature extraction for the proposed image enhancement scheme. Experiment results prove that the novel feature extraction is free from some artifacts on the edges such as blocking artifacts, “gradient reversal”, and pseudo contours. Compared with other enhancement methods, the proposed method achieves the best performance in infrared image enhancement.     

Enhancing the capabilities of binary phase only filter

In this paper, we demonstrate that the capabilities of a binary phase-only filter (BPOF) can be enhanced to identify targets irrespective of rotation, scale or the imaging spectral band by utilizing the concept of log-polar transform and image fusion. Till date, BPOFs have been considered to be the simplest of all filters and incapable of identifying distorted images or images of different spectral bands like the visible or infrared (IR) bands. The novelty of this work lies in the approach adopted to demonstrate that a BPOF is equally capable of distortion-invariance like any other distortion-invariant complex matched filter. This is done by suitably fusing the images of visible and IR bands and then taking the log-polar transformation of the fused image to synthesize the BPOF. A single BPOF is thus sufficient to identify (0-360)° in-plane rotated images, (50-190)% scaled images, combination of rotation and scale changes of the target, noisy image of both the visible and IR spectral bands. A further enhancement of the correlation peak intensity (CPI) is achieved by modifying this BPOF with Mexican-hat wavelet. The designed filter was implemented in the hybrid digital-optical correlation scheme. Correlation peak intensity and peak correlation energy (PCE) have been calculated as metrics of goodness of the proposed approach. Experimental results have been presented.     

Infrared and visible image fusion method based on saliency detection in sparse domain

Infrared and visible image fusion is a key problem in the field of multi-sensor image fusion. To better preserve the significant information of the infrared and visible images in the final fused image, the saliency maps of the source images is introduced into the fusion procedure. Firstly, under the framework of the joint sparse representation (JSR) model, the global and local saliency maps of the source images are obtained based on sparse coefficients. Then, a saliency detection model is proposed, which combines the global and local saliency maps to generate an integrated saliency map. Finally, a weighted fusion algorithm based on the integrated saliency map is developed to achieve the fusion progress. The experimental results show that our method is superior to the state-of-the-art methods in terms of several universal quality evaluation indexes, as well as in the visual quality.     

利用匹配区域的纹理特征改善重构三维图像的视觉质量

研究了计算机重构三维图像时分辨率低的问题,提出一种改善3D计算机全景重构图像的视觉质量的方法,该方法利用3D空间的物体部分在每个元素图像中形成的匹配区域的纹理特征,从两个相邻的元素图像中的匹配区域提取出多个像素,经过加权计算重构出相应的图像区域.该方法与传统的计算机重构方法相比,提高了图像分辨率,同时也消除了从每个元素图像中提取多个像素直接重构图像时存在的"像素块"效应,改善了重构图像的视觉质量.     

Corner detection and matching for infrared image based on double ring mask and adaptive SUSAN algorithm

Infrared (IR) image fusion is designed to fuse several IR images into a comprehensive image to boost imaging quality and reduce redundancy information, and image matching is an indispensable step. However, Conventional matching techniques are susceptible to the noise and fuzzy edges in IR images and it is therefore very desirable to have a matching algorithm that is tolerant to them. This paper presents a method for infrared image matching based on the SUSAN corner detection. To solve the problems of the traditional SUSAN algorithm including the fixed threshold of gray value difference and the failed detection of symmetry corners, an adaptive threshold extraction method is raised in this study. Furthermore, an attached double ring mask is used to improve the complex corner detection capability. A constraint condition and a principle of gravity are adopted to filtrate the candidate corners. The proposed method is qualitatively and quantitatively evaluated on IR images in the experiments. In comparison with other methods, better performance has been achieved.     

Digital holographic image fusion for a larger size object using compressive sensing

Digital holographic imaging fusion for a larger size object using compressive sensing is proposed. In this method, the high frequency component of the digital hologram under discrete wavelet transform is represented sparsely by using compressive sensing so that the data redundancy of digital holographic recording can be resolved validly, the low frequency component is retained totally to ensure the image quality, and multiple reconstructed images with different clear parts corresponding to a laser spot size are fused to realize the high quality reconstructed image of a larger size object. In addition, a filter combing high-pass and low-pass filters is designed to remove the zero-order term from a digital hologram effectively. The digital holographic experimental setup based on off-axis Fresnel digital holography was constructed. The feasible and comparative experiments were carried out. The fused image was evaluated by using the Tamura texture features. The experimental results demonstrated that the proposed method can improve the processing efficiency and visual characteristics of the fused image and enlarge the size of the measured object effectively.     

基于优势特征图像融合的水下光学图像增强

针对水下光学图像颜色失真、非均匀光照、对比度低的问题,提出基于优势特征图像融合的水下光学图像增强算法.首先,提出改进的暗通道先验算法去除退化图像中的不均匀浑浊并均衡色彩;其次,对颜色校正图像分别使用基于加权分布的自适应伽玛校正算法和限制对比度自适应直方图均衡-同态滤波算法,增强颜色校正图像对比度并使其亮度均衡;最后,定义三幅融合图像即颜色校正图像、亮度均衡图像、对比度增强图像的关联权重图,通过多尺度融合算法获得融合图像.与单一预处理算法只能解决对应的退化现象相比,该算法对单幅退化图像进行多算法处理,得到三幅优势特征图像,通过不同权重的组合最大程度地将各优势特征相结合,得到的综合效果远超各单一算法优化效果,不再局限于解决颜色失真等单一问题.将本文算法与现有算法在主观评价和客观评价两方面进行实验对比,结果表明,该算法可以有效平衡水下图像的色度、饱和度及清晰度,视觉效果接近自然场景下的图像.     

Image degradation characteristics and restoration based on regularization for diffractive imaging

The diffractive membrane optical imaging system is an important development trend of ultra large aperture and lightweight space camera. However, related investigations on physics-based diffractive imaging degradation characteristics and corresponding image restoration methods are less studied. In this paper, the model of image quality degradation for the diffraction imaging system is first deduced mathematically based on diffraction theory and then the degradation characteristics are analyzed. On this basis, a novel regularization model of image restoration that contains multiple prior constraints is established. After that, the solving approach of the equation with the multi-norm coexistence and multi-regularization parameters (prior’s parameters) is presented. Subsequently, the space-variant PSF image restoration method for large aperture diffractive imaging system is proposed combined with block idea of isoplanatic region. Experimentally, the proposed algorithm demonstrates its capacity to achieve multi-objective improvement including MTF enhancing, dispersion correcting, noise and artifact suppressing as well as image’s detail preserving, and produce satisfactory visual quality. This can provide scientific basis for applications and possesses potential application prospects on future space applications of diffractive membrane imaging technology.     

Multifocus image fusion scheme using focused region detection and multiresolution

To solve the fusion problem of the multifocus images of the same scene, a novel algorithm based on focused region detection and multiresolution is proposed. In order to integrate the advantages of spatial domain-based fusion methods and transformed domain-based fusion methods, we use a technique of focused region detection and a new fusion method of multiscale transform (MST) to guide pixel combination. Firstly, the initial fused image is acquired with a novel multiresolution image fusion method. The pixels of the original images, which are similar to the corresponding initial fused image pixels, are considered to be located in the sharply focused regions. By this method, the initial focused regions can be determined, and the techniques of morphological opening and closing are employed for post-processing. Then the pixels within the focused regions in each source image are selected as the pixels of the fused image; meanwhile, the initial fused image pixels which are located at the focused border regions are retained as the pixels of the final fused image. The fused image is then obtained. The experimental results show that the proposed fusion approach is effective and performs better in fusing multi-focus images than some current methods.