LPGAN: A LBP-based Proportional Input Generative Adversarial Network for Image Fusion

1. Introduction

Image fusion aims to merge or combine images captured with different sensors or camera settings to generate a greater quality composite image [1]. It is crucial for many applications in image processing[2,3], computer vision [4,5], remote sensing [6], and medical image analysis [7].

In image processing field, visible images and infrared images are generated by sensors with different sensitivities to light in different wavelength bands. The images of different bands contain different information. Each kind of image can only focus on a given operating range and environmental condition, and it is difficult to receive all the necessary information for object detection or the object or scene classification [8]. Due to the strong complementarity between them, it is a feasible way to improve visual understanding by fusing them. The fused image can combine the characteristics of different modal images to generate an image with rich details and significant contrast, an example is shown in Figure 1. Therefore, for the full exploitation of multi-modal data, advanced image fusion has been developed rapidly in the last few years.

The key to multi-modal image fusion is effective image information extraction and appropriate fusion principles [9]. Traditional research has focused on multi-scale transform-based methods [10,11], sparse representation-based methods [12,13], subspace-based methods [14], saliency-based methods [15], hybrid methods [16,17] and other fusion methods [18,19]. However, the performance improvement of artificially designed feature extraction and fusion rules is limited.

With the widespread application of deep learning, deep learning-based fusion methods have achieved rapid progress, showing advantages over conventional methods and leading to state-of-the-art results [20,21]. Although these algorithms have achieved positive results under most conditions, there are still some shortcomings that need to be improved:

• The extraction of source image feature information is incomplete. Most image fusion algorithms cannot achieve good structural similarity and retain rich detailed features at the same time due to the incomplete extraction of feature information [22,23,24].
• The mission objectives and network structure do not match. The same network is employed to extract features while ignoring the feature distribution characteristics of different modal images, resulting in the loss of meaningful information [24,25].
• Improper loss function leads to missing features. In the previous methods [22,23], only the gradient is used as a loss to supervise the extraction of detailed features and texture features, which makes the network feature extraction incomplete.

To overcome the above challenges, a completely deep learning-based method, LPGAN, is proposed to achieve image fusion. Firstly, we innovatively introduce local binary patterns (LBP) into the network to effectively extract the texture features of the source image. The LBP [26] can accurately describe the local texture features of the image, and because of its strong robustness to illumination, the introduction of LBP not only enables the network to have stronger feature extraction capabilities but also strong anti-interference capabilities. Sencondly, we introduce a pseudo-siamese network for the generator to extract the detailed features and contrast features in the feature extraction stage. It is worth mentioning that, the improved generator inputs infrared and visible images at different ratios. In some works [27,28], the source image is concatenated in a 1:2 ratio as the input of the network, but we found through experiments that a ratio of 1:4 can achieve better results. Extensive experiments on the publicly available TNO dataset and CVC14 dataset [29] show that the proposed method achieves the state-of-the-art performance. We also test the universality of LPGAN through the fusion of RGB and infrared images on the RoadScene dataset [25]. Finally, the proposed method can also be applied to multi-spectral remote sensing image fusion, and the expansion experiment reveals the advantages of our LPGAN compared to other methods.

The contributions of our work are as three-fold:

1)

We introduce LBP into the network and design a new content loss for the generator, which enables the model to extract features completely and reduce image distortion.

2)

Taking into account the characteristics of the information distribution of the source images, we propose a 1:4 scale input in the feature extraction stage.

3)

We design a pseudo-siamese network to extract feature information from source images. It fully considers the differences in the imaging mechanism and image features of the source images, encouraging the generator to preserve more features in source images.

We organized the remainder of this paper as follows. We first briefly review some work related to our method in Section II. Then, we provide the overall framework, network architecture and loss function of the proposed LPGAN in Section III. In Section IV, we show and analyse the experimental results of the proposed method and the competitors. In Section V, we give a discussion about our method. Finally, we make a conclusion of this paper in Section VI.

2. Related Work

Image fusion tasks can be divided into five types: infrared-visible image fusion, multi-focus image fusion, multi-exposure image fusion, remote sensing image fusion and medical image fusion. In this section, we briefly introduce several existing deep learning-based image fusion methods and some basic theories of cGAN and LBP.

2.1. Deep Learning-Based Image Fusion

Many deep learning based image fusion methods have been proposed in the last five years and have achieved promising performances. In some methods, the framework of deep learning is combined with traditional methods to solve image fusion tasks. Representatively, Liu et al. [30] proposed a fusion method based on convolutional sparse representation (CSR). The method employs CSR to extract multi-layer features and uses the features to reconstruct the image. Later they proposed a convolutional neural network (CNN)-based method for multi-focus image fusion tasks [31]. They use image patches containing different features as input to train the network and obtain a decision map, and then directly use the map to guide the image fusion. Deep learning is also used by some algorithms to diversify the extraction of image features. In [32], a model based on the multi-layer fusion strategy of the VGG-19 model was proposed. The method decomposes the source image into two parts: one part contains the low-frequency information of the image and the other part contains the high-frequency information with detailed features. This strategy can retain the deep features of the detailed information. In addition, to make the generated images more realistic, PSGAN [33] handles the remote sensing image fusion task by using a GAN to fit the distribution of high-resolution multi-spectral images. However, it still requires artificially constructing the ground-truth to train the model.

The above methods only apply the deep learning framework in some parts of the fusion process. In other methods, the entire image fusion process uses a deep learning framework. For instance, Prabhakar et al. [34] proposed an unsupervised model for multi-exposure image fusion named DeepFuse in 2017. The model consists of an encoder, a fusion layer and a decoder. The parameter sharing strategy is adopted to ensure that the feature types extracted from the source images are the same, which facilitates subsequent fusion operations and reduces the parameters of the model. Based on DeepFuse, Li et al. [24] improved the method by applying dense blocks and proposed a new image fusion method called DenseFuse. They utilize no-reference metrics as the loss function to train the network and achieve a high-quality performance. Since image fusion tasks usually lack ground-truth and are generative tasks, Ma et al. [22] proposed a GAN-based method to fuse infrared and visual images. The network uses a generator to fuse images and a discriminator to distinguish the generated image from the visible image, achieving a state-of-the-art performance. However, it is easy to lose infrared image information. To avoid the above problem, Ma et al. [23] used dual discriminators to encourage the generator, and the method achieved a better performance. Analogously, Ma et al. [35] applied a dual-discriminator architecture in remote sensing image fusion and proposed an unsupervised method based on GAN, termed PanGAN. The method establishes adversarial games to preserve the rich spectral information of multi-spectral images and the spatial information of panchromatic images. By considering the different characteristics of different image fusion tasks, Xu et al. [25] performed continual learning to solve multiple fusion tasks for the first time and proposed a unified unsupervised image fusion network named U2Fusion, which could be applied to a variety of image fusion tasks, including multi-modal, multi-exposure, and multi-focus cases.

However, the abovementioned works still have three drawbacks:

• Due to the lack of ground-truth, the existing methods usually supervise the work of the model by adopting no-reference metrics as the loss function. However, only the gradient is used as the loss to supervise the extraction of the detailed features, and the texture information is always ignored.
• They ignore the information distribution of the source images, i.e., the visible image has more detailed information and the infrared image has more contrast information.
• These methods all use only one network to extract features from infrared images and visible images, ignoring the difference in imaging mechanisms between these two kinds of images.

To address these problems, a new content loss function is designed using LBP to effectively utilize the texture information of the source images, which also improves the anti-interference ability of our method. Then, according to the distribution characteristics of the feature information of the source images, we concatenate the source images in fixed proportions in the feature extraction stage. Finally, considering the characteristics of infrared image and visible image, we design a pseudo-siamese network to extract detailed features and contrast features respectively.

2.2. Generative Adversarial Networks

GAN is a framework for unsupervised distribution estimation via an adversarial process, proposed by Goodfellow [36] in 2014. The GAN simultaneously trains two models: a generative model G that captures the data distribution and a discriminative model D that estimates the probability that a sample comes from the training data rather than G. The GAN establishes an adversarial game between a discriminator and a generator, the generator tries to continuously generate new samples to fool the discriminator, and the discriminator aims to judge whether a sample is real or fake. Finally, the discriminator can no longer distinguish the generated sample. Assuming that the real data obey the specific distribution $P_{data}$, the generator is dedicated to estimating the distribution of real data and producing the fake distribution $P_G$ that approaches the real distribution $P_{data}$. D and G play the following two-player minimax game with value function $V\left(G,D\right)$:

$$\begin{array}{c}\underset{G}{min}\underset{D}{max}{V}_{GAN}\left(G,D\right)={\mathbb{E}}_{x\sim P_{data}}\left[logD\left(x\right)\right]\hfill \\ \hfill +{\mathbb{E}}_{x\sim P_G}\left[log\left(1-D\left(x\right)\right)\right],\end{array}$$

(1)

$\mathbb{E}$ is the average operation. Due to the adversarial relationship, the generator and the discriminator promote each other in continuous iterative training, and the capabilities of the two are continuously improved. The sample distribution generated by the generator approaches the distribution of the real data. When the similarity between the two is high enough, the discriminator cannot distinguish between the real data and the fake data, and the training of the generator is successful.

GANs can be extended to a cGAN if we add some extra information that could be any kind of auxiliary information as a part of the input. We can perform conditioning by feeding the extra information as an additional input layer, and this model is defined as a cGAN [37]. The formulation between G and D of cGAN is as follows:

$$\begin{array}{c}\hfill \underset{G}{min}\underset{D}{max}{V}_{GAN}\left(G,D\right)={\mathbb{E}}_{x\sim P_{data}}\left[logD\left(x\left|y\right.\right)\right]\\ \hfill +{\mathbb{E}}_{x\sim P_G}\left[log\left(1-D\left(x\left|y\right.\right)\right)\right].\end{array}$$

(2)

Standard GANs consist of a single generator and only one discriminator. In order to generate higher quality samples in fewer iterations, Durugkar et al. [38] proposed the Generative Multi-Adversarial Network (GMAN), a framework that extends GANs to multiple discriminators. Inspired by GMAN, the structure of multi-adversarial network is applied for dealing with different tasks, such as PS2MAN [39] and FakeGAN [40], etc. PS2MAN considered the photo-sketch synthesis task as an image-to-image translation problem and explored the multi-adversarial network (MAN) to generate high-quality realistic images. Novelly, FakeGAN first adopted GAN for a text classification task. The network provided the generator with two discriminators, which avoided the mod collapse issue and provided the network with high stability. One discriminator is trained to guide the generator to produce samples similar to deceptive views, and the other one aims to distinguish deceptive views from data.

Image fusion is a generating task that integrates two images of different characteristics. GAN is a network suitable for unsupervised generative tasks. Therefore, we adopt GAN as the framework of our method. To preserve detailed information and contrast information of two source images completely, we employ dual discriminators to improve the quality of our fusion results.

2.3. Local Binary Patterns

LBP is an operator used to describe the local texture features of images, which is gray scale invariant and can be easily calculated by comparing the center value with its $3\times 3$ neighbors [26]. Although the original LBP can effectively extract the texture features of the image and has strong robustness to illumination, it cannot cope with the scaling and rotation of the image. To address this problem, Ojala et al. [41] in 2002 proposed an improved LBP with scale invariance and rotation invariance. The improved LBP compares the center pixel with pixels on a fixed radius, which changes as the image is scaled, thus achieving scale invariance of LBP features. At the same time, the minimum value of the encoded binary number is taken to achieve the rotation invariance of the LBP feature. LBP is used in many fields of machine learning. In [42], Zhao et al. applied it to recognize the dynamic textures and extend their approach to deal with specific dynamic events such as facial expression recognition. Maturana et al. [43] also proposed a LBP-based face recognition algorithm. LBP has also been applied in the field of gender recognition and was once the most effective method in this field. Tapia et al. [44] extract the iris features of the human eyes through the effective texture feature extraction capability of LBP, and perform gender recognition based on the extracted features, achieving the state-of-the-art performance of gender recognition at that time.

Although LBP was widely used in the past, it has rarely been mentioned in recent years. Since image fusion has high requirements on the detailed features and structural similarity of the fused image, the existing algorithms cannot achieve the above two points simultaneously. We believe that LBP can help the network to extract lower-level detailed features while keeping the fused image with a high structural similarity to the source images. Because the source images for image fusion are highly registered, there is no need to consider the rotation and scaling of the image, so we use the original LBP to extract the texture features of the source images.

3. Proposed Method

In this section,we introduce the proposed LPGAN in detail. Firstly, we describe the overall framework of LPGAN, and then we provide the network architectures of the generator and the discriminators. Finally, the loss function is designed.

3.1. Overall Framework

The overall framework of the proposed LPGAN is sketched in Figure 2. It is a dual-discriminator cGAN. Visible images have rich detailed information that is saved through gradient and texture, and infrared images save significant contrast information through pixel intensity. The goal of the infrared-visible image fusion task is to generate a new image with rich detailed information and significant contrast information, which is essentially an unsupervised generation task. GANs have significant advantages in dealing with such problems, so this research chooses GAN network as the basic framework. Given a visible image $I_{vis}$ and an infrared image $I_{ir}$, the goal of GAN applied to image fusion is to train a generator G to produce a fused image $I_f$ , and then $I_f$ is realistic enough to fool the discriminator. Due to the differences between infrared and visible images, we introduce a GAN with two discriminators to ensure that the feature extraction of infrared images and visible images is complete.

We design a pseudo-siamese network to extract the detailed features and contrast features. Considering that visible images still have some contrast information, infrared images also have some detailed information. We design different input ratios for different encoders to concatenate the visible image $I_{vis}$ and infrared image $I_{ir}$ in the channel dimension. Specifically, the ratio of the detailed feature extraction path and the contrast feature extraction path is set to 4:1 ($I_{vis}$ : $I_{ir}$) and 1:4 ($I_{vis}$ : $I_{ir}$), respectively. Then, the concatenated images are fed into the generator G, and the output of G is a fused image $I_f$. After that, LBP distributions of source images and $I_f$ are calculated and the loss of LBP used to supervise G to extract texture features is obtained. Simultaneously, we design two adversarial discriminators, $D_{vis}$ and $D_{ir}$. $D_{vis}$ and $D_{ir}$ generate scalars based on the input image to distinguish the generated image and real data. $D_{vis}$ is trained to generate the probability that the image is a real visible image, while $D_{ir}$ is trained to estimate the probability of the image belonging to the real infrared images.

3.2. Network Architecture

3.2.1. Generator Architecture

The generator consists of a feature extraction network and a feature reconstruction network, as shown in Figure 3. The feature extraction network takes the form of a pseudo-siamese network, which is divided into gradient path and intensity path for information extraction. The process of feature reconstruction is performed in a decoder, and the output is the fused image, which has the same resolution as the source images.

In the feature extraction stage, we propose a pseudo-siamese network with two encoders. Inspired by DenseNet [45], to mitigate the vanish of gradient, remedy feature loss and reuse previously computed features, both encoders are densely connected and have the same network structure, but the parameters of them are different. In each path of feature extraction, the encoder consists of 4 convolutional layers. The $3\times 3$ convolutional kernel is adopted in each layer, and all strides are set to 1 with a batch normalization (BN) and a ReLU activation function to speed up the convergence and avoid gradient sparsity [46]. To fully extract the information, we concatenate 4 visible images and 1 infrared image as input in the gradient path, as well as 4 infrared images and 1 visible image as input in the intensity path. After that, the outputs of the two paths are concatenated in the channel dimension. The final fusion result is generated by a decoder. The decoder is a 4-layer CNN, and the parameter settings of each layer are shown in the bottom right sub-figure of Figure 3.

3.2.2. Discriminator Architecture

The architecture of discriminator $D_{vis}$ and discriminator $D_{ir}$ adopt the same structure. The discriminator is a simple 4-layer convolution neural network, which is shown in Figure 4. In the first 3 layers, the $3\times 3$ filter is adopted in each convolution layer, and the stride is set to 2. BN and ReLU activation function are followed in each convolution layer. In the last layer, the full connection and the tanh activation function are employed to generate the probability of the input image belonging to the real data.

3.3. Loss Function

We adopt two types of loss, loss ${\mathcal{L}}_G$ and loss ${\mathcal{L}}_D$, to guide the parameter optimization of G and D.

3.3.1. Loss Function of Generator

The loss function of G consists of two parts, i.e., the content loss ${\mathcal{L}}_{con}$ and the adversarial loss ${\mathcal{L}}_{adv}$ :

$${\mathcal{L}}_G=\mu {\mathcal{L}}_{con}+{\mathcal{L}}_{adv},$$

(3)

where ${\mathcal{L}}_G$ is the total loss and $\mu$ is a parameter to strike a balance between ${\mathcal{L}}_{con}$ and ${\mathcal{L}}_{adv}$. As the thermal radiation and texture details are mainly characterized by pixel intensities and gradient variation [18], we design four loss functions to guide G to preserve the gradient and texture information contained in the visible image and contrast information of the infrared image and reduce image distortion. We employ the L1 norm to constrain the fused image to retain similar gradient variation with the visible image. The calculation of gradient loss is as follows:

$${\mathcal{L}}_{gradient}={∥\nabla I_f-\nabla I_{vis}∥}_1,$$

(4)

where ∇ is the unification of the horizontal and vertical gradients of the image. It is calculated as:

$$\nabla =\sqrt{{\left[I(i+1,j)-I(i-1,j)\right]}^2+{\left[I(i,j+1)-I(i,j-1)\right]}^2},$$

(5)

where $I(i,j)$ represents the pixel value of the image at $(i,j)$. Contrast information is mainly saved by the pixel intensities of the image. Therefore, the Frobenius norm is applied to encourage the fused image to exhibit pixel intensities similar to those of the infrared image, and the contrast loss is calculated as:

$${\mathcal{L}}_{intensity}=\frac{1}{WH}{∥I_f-I_{ir}∥}_F^2,$$

(6)

where W and H are the width and height of the image.

To prevent image distortion, we use a structural similarity loss ${\mathcal{L}}_{SSIM}$ to constrain the fusion of the generator. The loss ${\mathcal{L}}_{SSIM}$ is obtained by this equation:

$${\mathcal{L}}_{SSIM}=\frac{1}{2}[(1-SSIM(I_f,I_{vis}))+(1-SSIM(I_f,I_{ir}))],$$

(7)

where $SSIM(·)$ reflects the structural similarity of two images [47]. Visible images have rich detailed and texture information, which greatly improves the efficiency of tasks, such as target detection. In the previous algorithms, only the gradient is used as a loss to supervise the extraction of texture features. In this paper, we innovatively introduce LBP into the loss function to improve the extraction of texture features.

The formulation of ${\mathcal{L}}_{LBP}$ is shown as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{LBP}=\frac{1}{L}({∥LBP\left(I_f\right)-LBP\left(I_{vis}\right)∥}_1,\end{array}$$

(8)

where $LBP(·)$ represents the operation of calculating the LBP features of the image, and L is the feature vector length of $LBP(·)$. The calculation of $LBP(·)$ is defined as follows:

$$LBP\left(I\right)=concat[lbp\left(cel{l}_1\right),...,lbp\left(cel{l}_{36}\right)],$$

(9)

where $cel{l}_i$ is a $21\times 21$ image patch, and there are 16 cells in image I, as shown in Figure 5. $lbp\left(cell{}_i\right)$ aims to calculate the LBP feature of $cell{}_i$. To obtain $lbp\left(cell{}_i\right)$, we first calculate the LBP value of each pixel in $cell{}_i$ according to the method proposed in [26]. After calculating the LBP value of each pixel, a 256-dimensional vector is used to represent the LBP feature of the cell, which is the final result of $lbp(·)$. As shown in Figure 5, the result of $LBP(·)$ is the concatenation of $lbp\left(cel{l}_i\right),(i=1,...,16)$, and L in Eq. 9 is $4,096$ here.

To summarize, the proposed ${\mathcal{L}}_{con}$ consists of 4 parts as shown in Eq. 10:

$${\mathcal{L}}_{con}=\alpha {\mathcal{L}}_{gradient}+\beta {\mathcal{L}}_{intensity}+\gamma {\mathcal{L}}_{SSIM}+\lambda {\mathcal{L}}_{LBP},$$

(10)

where $\alpha$, $\beta$, $\gamma$ and $\lambda$ are parameters used to control the trade-off between four terms.

The adversarial loss ${\mathcal{L}}_{adv}$ in Eq.3 denotes the sum of two adversarial losses between the generator G and two discriminators, which can be formulated as:

$${\mathcal{L}}_{adv}=\mathbb{E}\left[-log\left(1-D_{vis}\left(I_f\right)\right)\right]+\mathbb{E}\left[-log\left(1-D_{ir}\left(I_f\right)\right)\right],$$

(11)

where $I_f$ denotes the fused image, $D_{vis}\left(I_f\right)$ denotes the probability that $I_f$ belongs to the real visible image and $D_{ir}\left(I_f\right)$ denotes the probability that $I_f$ is an infrared image.

3.3.2. Loss Function of Discriminators

The visible and infrared images contain rich texture details and contrast information. We establish an adversarial game between a generator and two discriminators for the result of the generator to match more with the distribution of the real data. Formally, the loss functions of discriminators are defined as follows:

$${\mathcal{L}}_{D_{vis}}=\mathbb{E}\left[-log{D}_{vis}\left(I_{vis}\right)\right]+\mathbb{E}\left[-log\left(1-D_{vis}\left(I_f\right)\right)\right],$$

(12)

$${\mathcal{L}}_{D_{ir}}=\mathbb{E}\left[-log{D}_{ir}\left(I_{ir}\right)\right]+\mathbb{E}\left[-log\left(1-D_{ir}\left(I_f\right)\right)\right].$$

(13)

4. Experiments

In this section, we evaluate our method on publicly available datasets: the TNO dataset, the CVC14 dataset, the RoadScene dataset and some multi-spectral images. First, we provide the detailed experimental configurations. Then, we compare the results of our methods with four state-of-the-art methods: FusionGAN [22], DenseFuse [24], DDcGAN [23] and U2Fusion [25] on the first two public datasets. We also verify the improvement of the network performance by LBP and 1:4 ratio input through ablation experiment. Third, we test our method on the RoadScene dataset by fusing the infrared image and RGB image. Finally, we apply our method to multi-spectral remote sensing images and compare it with above four algorithms.

4.1. Implementation

4.1.1. Dataset

We validate our method on three classical fusion image datasets: the TNO dataset, the CVC14 dataset and the RoadScene dataset. Some multispectral images are also selected to test it.

The TNO dataset is the most commonly used infrared visible image Human Factors dataset. It contains multi-spectral images of different scenarios registered with different multi-band camera systems [48].

The CVC14 dataset is committed to promoting the development of autonomous driving technologies [49,50]. It consists of two sets of sequences: the day set and night set. The day set includes $8,821$ images, the night set includes $9,589$ images, and all images have a $640\times 471$ resolution.

The RoadScene dataset is a new image fusion dataset that has 221 infrared and visible image pairs. The images of the dataset are all collected from naturalistic driving videos, including roads, pedestrians, vehicles and other road scenes.

The multi-spectral remote sensing images used in this paper are recorded under the USA Airborne Multisensor Pod System (AMPS) program and include a large number of industrial, urban and natural scenes from a number of geographical locations captured by two hyper-spectral airborne scanners [51].

In the training stage, we adopt the overlapping cropping strategy to expand the dataset. Thirty-six infrared and visible image pairs of TNO are cropped into $22,912$ patch pairs with $84\times 84$ pixels. The $84\times 84$ visible and infrared image patches are used as source images to train the generator G and as labels to encourage the discriminators. For testing, we select 14, 26, 5 and 29 image pairs from the TNO, CVC14, RoadScene datasets and multi-spectral images, respectively.

4.1.2. Training Details

As mentioned in Section 3, parameters $\mu$, $\alpha$, $\beta$, $\gamma$ and $\lambda$ are used to control the balance of loss functions. We set $\mu =0.6$, $\alpha =0.2$, $\beta =0.03$, $\gamma =500$ and $\lambda =0.5$. The initial learning rate and decay rate are set to 0.0002 and 0.75 to train the model, and RMSprop and SGD are adopted as the optimizers of the generator and discriminators, respectively. All experiments are conducted on a desktop with 2.30 GHz Intel Xeon CPU E5-2697 v4, NVIDIA Titan Xp, and 12 GB memory.

4.1.3. Metrics

In the test stage, we evaluate our method under 8 evaluation metrics to comprehensively compare it with the current methods.

An image fusion qualitative assessment mainly starts from the human visual system and judges the fusion effect according to the task goal. The goal of infrared-visible image fusion is to preserve the detailed and texture features of visible images and the contrast features of infrared images as much as possible. Conversely, quantitative evaluation comprehensively reflects the effect of image fusion through a variety of evaluation metrics.

In this paper, we select 8 metrics to use to evaluate our LPGAN and 4 other state-of-the-art methods. The metrics are standard deviation (SD) [52], average gradient (AG) [53], spatial frequency (SF) [54], mutual information (MI) [55], entropy (EN) [56], peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM) [57], and visual information fidelity (VIF) [58]. $SD$ reflects the distribution of pixel values and contrast information. The larger the $SD$ is, the higher the contrast and the better the visual effect. $AG$ quantifies the gradient information of an image and reflects the amount of image details and textures. The larger $AG$ is, the more detailed information the image contains and the better the fusion effect. $SF$ is a gradient-based metric that can measure the gradient distribution effectively and reveal the details and texture of an image. The larger $SF$ is, the richer edges and texture details are preserved. $MI$ is a quality index that measures the amount of information that is transferred from source images to the fused image [55]. A larger $MI$ represents more information that is transferred from source images to the fused image, which means better fusion performance. $EN$ is a metric to measure the amount of information contained in the image, and the larger the $EN$ value is, the more informative. $PSNR$ is a metric reflecting the distortion and anti-interference ability by the ratio of peak value power and noise power [23]. A large $PSNR$ indicates that little distortion occurred and there is a strong anti-interference ability. $SSIM$ is used to measure the structural similarity between two images and consists of three components: loss of correlation, loss of luminance and contrast distortion. The product of the three components is the assessment result of the fused image [9]. We calculate the average $SSIM$ between the fused image and two source images as the final result. A larger value of $SSIM$ indicates that more structural information is maintained. $VIF$ is consistent with the human visual system and is applied to measure information fidelity. The larger $VIF$ is, the better the visual effect and the less distortion there is between the fused image and source images.

4.2. Results on the TNO Dataset

4.2.1. Qualitative Comparison

We provide 6 image pairs to report some intuitive results on the fusion performance, as shown in the Figure 6. Compared with other methods, our LPGAN has 3 advantages. First, the proposed method maintains the high-contrast characteristics of infrared images, as shown in the third and fourth examples, which proves to be effective for automatic target detection and location. Taking the third result as an example, only our LPGAN and FusionGAN can clearly distinguish the front and top of the pipeline and clearly see the edges of it, but FusionGAN poorly preserves the details of other areas. Second, our LPGAN can preserve rich texture details of visible images, which is beneficial for accurate target recognition. As shown in the first example, only the result of LPGAN can distinguish the detailed and texture features of the ground. Finally, our results all have a rich pixel intensity distribution, which means our results have more information and are consistent with the human visual system.

As shown in Figure 6, DenseFuse and U2Fusion cannot preserve the thermal information well in the infrared image. The third group of results shows the brightness of the buildings and pipelines is considerably lower than that of the infrared image. Conversely, FusionGAN and DDcGAN cannot retain the detailed and texture information of visible images. In the second set of results, FusionGAN cannot retain the detailed features of the chimney well, while DDcGAN hardly retains the detailed features of the human. In contrast, although LPGAN tends to retain the detailed features of visible images, it still preserves important contrast information from the infrared images. As shown in the first column of Figure 6, in our result, the details of the ground can be clearly distinguished, while the thermal information of the infrared image is also well preserved.

4.2.2. Quantitative Comparison

Figure 7 shows the results of all of the examined methods on 14 tested image pairs of the TNO dataset. Our LPGAN can generate the largest average values on $SD$, $AG$, $SF$, and $MI$ and the second largest values on $EN$. In particular, our LPGAN achieves the best values of $SD$, $SF$, $MI$ and $EN$ on 6, 6, 7 and 6 image pairs, respectively. For $PSNR$, $SSIM$ and $VIF$, our method ranks third, but the gaps with the top-ranked methods are very small. These results demonstrate that our method is able to preserve the best edges and texture details and contains the highest contrast information. Our results also contain rich information. The value on $EN$ is only less than DDcGAN, but our performance on $MI$ is better, meaning that DDcGAN has more false information than our method. In addition, our method can reduce noise interference very well and has a strong correlation with the source images. Finally, although DenseFuse and U2Fusion have larger values of $SSIM$, our method achieves a better balance between feature information and the visual effect. From the $VIF$, it can be seen that the results of our method have a good visual effect.

4.3. Results on the CVC14 Dataset

To evaluate the effectiveness of our proposed method, we conduct an experiment on the CVC14 dataset. Twenty-six image pairs are selected from different scenes for evaluation.

4.3.1. Qualitative Comparison

We perform a qualitative comparison on 5 typical image pairs, as shown in the first two rows of Figure 8, to demonstrate the characteristics of our method. The different light and dark changes in the results indicate that only DDcGAN and our LPGAN have high contrast information. For example, in the first and second images, the pixel intensity distributions of the cars, buildings, roads and people in the results of these two methods are abundant, but the results of other methods are not as obvious. Nevertheless, our results preserve more light information, as shown in the last group of results, meaning that LPGAN can extract more features from the source images. In terms of detail information preservation and visual effects, DDcGAN has more artifacts in the image due to the pursuit of high contrast, while the results of DenseFuse and U2Fusion have poor image visual effects due to less contrast. In contrast, our method can retain rich edges and texture details, while avoiding blurring and recognition difficulties due to darker colors, as shown in the third group.

4.3.2. Quantitative Comparison

As shown in Figure 9, 26 testing image pairs of the CVC14 dataset are selected to further display quantitative comparisons of our LPGAN and the other examined methods. Our LPGAN still achieves the largest mean values on $AG$, $SF$, and $MI$. In particular, our LPGAN achieves the largest values of $SF$, $AG$, and $MI$ on 21, 15 and 15 image pairs. On $SF$ and $MI$, our results are 8.3% and 7.0% higher than the second place, respectively. For the metric $SD$ and $EN$, our LPGAN can also achieve comparable results and only follows behind DDcGAN. However, the lower values of $MI$ and $PSNR$ indicate that there is more noise and fake information in the DDcGAN results. For $PSNR$, $SSIM$ and $VIF$, all algorithms performed very well, with small gaps. The results demonstrate that LPGAN can not only extract rich detailed information from visible images but also retain important contrast information from infrared images.

4.4. Ablation Study

Because images of different modalities have different information distributions, we adopt a 1:4 ratio input to improve the ability of network feature extraction. At the same time, in order to ensure that the fused images have rich details and high structural similarity, we introduce LBP into the loss function to guide the optimization of the network. In this work, to evaluate the effect of a 1:4 ratio input and LBP, we train six models with exactly the same parameter settings on the TNO dataset according to the ratio of the input and whether LBP is used. In Figure 10, we select a set of experimental results to show the differences between the six models.

4.4.1. The effect of LBP

We use an ablative comparison by removing the LBP. As shown in Figure 10, given the same ratio of inputs, the fusion results of LPGAN trained with ${\mathcal{L}}_{LBP}$ contain more detailed information and are more in line with human-visible systems. In the second row of the figure, the images generated by LPGAN trained with ${\mathcal{L}}_{LBP}$ obviously have sharper outlines, and there is more detailed background information. In addition, without using ${\mathcal{L}}_{LBP}$, the results have less contrast and more distortion. Through Table 1, we can find that after adding LBP, the performance of all models are improved in almost all evaluation metrics, especially in SD, AG and SF. This shows that after adding LBP, the ability of the model to extract detailed features are indeed enhanceed.

4.4.2. The effect of proportional input

The proportional input method can enhance the ability of the network to extract the feature information of different modal images, which is conducive to the effect of fusion. We explore its effectiveness of different ratios of input by setting the ratio as 1:2, 1:3 and 1:4. Since the models using LBP have better performance, we compare the models using LBP. As shown in Table 1, the last model (1:4 w/LBP) reaches the best in three of the eight evaluation metrics, and three reaches the second best value. The other two also has a small gap with the top value, especially in the key metrics such as SD, MI and SSIM, which are better than the first model (1:2 w/LBP) and the second model (1:3 w/LBP). The second model (1:3 w/LBP) achieves the best in the four evaluation metrics, but its performance in MI and SSIM is poor, indicating that its output results contain more false information, and as shown in Figure 10, the model with 1:4 scale has better visual effect compared with the other two, so we finally choose 1:4 as the input ratio of the network.

4.5. Additional Results for RGB Images and Infrared Images

Apart from grayscale image fusion, LPGAN can also be used in RGB-infrared image fusion task. As shown in Figure 11, we first convert the RGB image into the YCbCr color space. Then, we use the proposed LPGAN to fuse the luminance channel of the RGB image and infrared image. This is because the structural information is usually saved in the luminance channel. After that, the fused image is combined with chroma (Cb and Cr) channels and then converted into the RGB color space. In Figure 12, we select 5 sets of experimental results to show the effect of LPGAN. It can be seen from the results that LPGAN can fully extract feature information from infrared images and visible images and fuse them well. Taking the first image as an example, the license plate number in the visible light image is very fuzzy, but the model accurately fuses the two based on the feature information in the infrared image. Only DenseFusion, U2Fusion and the output results of our method can clearly see the license plate number of the vehicle. However, in the second group of experimental results, only the fusion result of our method can better preserve the color of the sky in the visible image, and only the result of our method can judge that it is daytime.

4.6. Multi-spectral Image Fusion Expansion Experiment

In this work, we apply our method to multi-spectral remote sensing image fusion and compare it with 4 state-of-the-art fusion algorithms.

We report 6 typical image pairs, as seen in Figure 13. The first two rows are multi-spectral images of two different bands, and both images are taken from the same scene and have the same resolution. The images in the first row have the same high contrast characteristics as the infrared images, and the images in the second row have the same feature of rich detailed information as the visible images. Thus, we follow the idea of infrared and visible image fusion to fuse these two remote sensing images.

Detailed features are the most important information in remote sensing images. DenseFuse, U2Fusion and our LPGAN can preserve it well, but our method performs better. In the third and fifth groups of results, only LPGAN exhibits subtle changes in ground details without producing artifacts, which is very important in small target detection tasks. In addition, it is obvious that only DDcGAN and our method can achieve a high contrast. For example, in the last set of results, DDcGAN and our method can clearly distinguish roads from background information, while for the other 3 methods it is more difficult. However, DDcGAN produces many artifacts in the process of image fusion, as shown in the fourth and fifth experimental results. In contrast, the results of our method all have clear images, no distortion or artifacts and are very consistent with the human visual system.

To evaluate the capability of LPGAN more objectively, we also conduct a quantitative assessment. Twenty-nine pairs of images are selected for testing and eight performance metrics are performed. For the characteristics of remote sensing images, we replace $VIF$ with the correlation coefficient (CC) [59]. The $CC$ expresses the degree to which the source image and fused image are related, and Pearson’s correlation is mostly used to measure the abovementioned correlation [60]. Table 2 shows the results of the quantitative comparisons. Our LPGAN achieves the best performances on $AG$ and $SF$ and achieves the second largest values in other metrics except $PSNR$. For $PSNR$, our method also shows comparable result and generates the third largest average value. DenseFuse is slightly better than our method in terms of $SSIM$, $MI$, and $CC$, but there is a large gap with our method in terms of $SD$, $AG$, and $SF$, which indicates that DenseFuse is not sufficient for detailed retention. U2Fusion and our method have a small gap in all metrics, but it only achieves better result than ours on $PSNR$. The largest values of $EN$ and $SD$ are achieved by DDcGAN, and LPGAN all ranks second. However, low values on $SSIM$, $MI$ and $CC$ indicate that there is considerable fake information in the results of DDcGAN, and DDcGAN cannot retain structural information from source images well.

5. Discussion

In this section, we discuss a key issue of image fusion and the effectiveness of our solution. We also introduce the limitations of our method and our future work.

For different characteristics and needs, many evaluation metrics for image fusion have been proposed. An excellent image fusion algorithm must not only have high-quality visual effects but also achieve good results on these metrics. During our experiments, we found that good visual effects conflict with some metrics, such as $SD$, $SF$ and $AG$. Specifically, an algorithm with a high $SSIM$ index generally has better visual effects, but its performance on $SD$, $SF$ and $AG$ will be poor, such as for FusionGAN, DenseFuse and U2Fusion. The reverse is also the same, such as for DDcGAN. Our LPGAN successfully avoids this problem. It not only performs well on $SD$, $SF$ and $AG$ but also performs well on $SSIM$. We use DDcGAN as an example to analyze the causes of the previous problem. In the process of image fusion, too much emphasis is placed on gradient intensity changes, and the gradient direction and the texture features of the source image are ignored, leading to poor visual effects. Based on this idea, we creatively introduce LBP into the loss function, and to preserve the spatial information of the image, we convert the calculated LBP into a 4096-dimensional vector. As shown in the results of the ablation experiment, the introduction of LBP can improve LPGAN in almost all aspects and successfully solve the abovementioned key problem.

Although LPGAN can achieve good performance on infrared-visible image fusion and multi-spectral image fusion tasks, avoiding the above problem, its visual effects and performance on $SSIM$ still need to be improved. In the future, we will try to consider image fusion from the perspective of decision-making and focus on the use of an attention mechanism to enable the network to perform fusion operations based on the information distribution of the source images. This is because in actual image fusion tasks, one source image is often of low quality. Therefore, we hope to adjust the fusion parameters adaptively by introducing an attention mechanism. Furthermore, we plan to apply our method in other image fusion tasks, such as medical image fusion, multi-focus image fusion and multi-exposure image fusion.

6. Conclusion

In this paper, we propose a novel GAN-based visible-infrared image fusion method, termed as LPGAN. It is an unsupervised end-to-end model. We adopt a cGAN as the framework and employ two discriminators, avoiding the mod collapse issue and providing the network with high stability. Simultaneously, considering the differences of imaging mechanisms and characteristics between visible images and infrared images, a pseudo-siamese network is used for generator to extract the detailed features and contrast features. We also set a 1:4 ratio input method according to the characteristics of different modal images to further improve the feature extraction capability of the network. In response to the existing problem, we innovatively introduce LBP into the loss function, which greatly improves the texture description ability and anti-interference ability of LPGAN. Compared with other four state-of-the-art methods on the publicly available TNO dataset, CVC14 dataset and ROADScene dataset, our method can achieve advanced performance both qualitatively and quantitatively. The experiment on a multi-spectral image fusion task also demonstrates that our LPGAN can achieve state-of-the-art performance.