CDFFusion: A Color-Deviation-Free Fusion Network for Nighttime Infrared and Visible Images

1. Introduction

Visible images usually contain rich texture detail information, but they are prone to loss of target information in complex scenes; infrared images, which are formed based on thermal radiation information, are not easily affected by harsh conditions, but they lack detailed descriptions of the scene. Therefore, infrared and visible image fusion (IVIF) can make full use of their complementary information and significantly improve the comprehensive perception ability of the scene. However, in current low-light environments with low visibility, most of the texture details in visible light images are obscured. The general approach to address this issue is to first preprocess the visible light image using a low-light enhancement method and then fuse it with the infrared image. However, this can cause color distortion in some areas, so how to organically combine low-light image enhancement with IVIF is a significant challenge.

Figure 1 shows the visible image, infrared image, the result processed by RFN-Net, and the result processed by LEDNet [1] preprocessing followed by RFN-Net. Firstly, previous IVIF methods fail to extract the information of visible images obscured at night (as shown in the red box in (c)). In contrast, fusing the visible image after enhancement preprocessing causes color distortion in some areas (as shown in the green box in (d), where white zebra crossings are rendered green).

Secondly, existing nighttime IVIF methods, while solving the above two problems, introduce new issues. Figure 2 displays the visible image, infrared image, and the results processed by DIVFusion, LEFuse, LENFusion, and the proposed method. It can be observed that DIVFusion and LEFuse overemphasize the difference between high and low gray values and focus on highlighting high-gray-value regions, which leads to two consequences: first, overexposure occurs in certain parts of the image (the lower left corners in (c) and (d)); second, severe block artifacts (mosaic effect, as shown in the red boxes in (c) and (d)) or false edges appear in non-edge or weak-edge regions. Although LENFusion avoids overexposure, it differs from the previous two methods by focusing on suppressing low-gray-value regions, which also results in two problems: in some weak-edge regions, the low-gray-value parts are severely distorted, leading to information loss (as shown in the red box in (e)); in other weak-edge regions, the colors of these low-gray-value parts are rendered extremely dark, causing unnatural false edges (not shown in the figure). Additionally, these methods only process the Y channel of visible images without handling the Cb and Cr channels, which changes the hue and saturation of the source images, resulting in varying degrees of color deviation.

To solve the above problems, this paper proposes a two-stage network for joint low-light image enhancement and image fusion without color deviation. It can alleviate overexposure and block artifacts while eliminating color deviation. Firstly, an encoder with a Feature Pyramid Network (FPN) structure is used to extract deep features of visible and infrared images respectively. Then, combined with the RetiNex theory, a decomposition network is designed at the feature level to separate the illumination component and reflectance component of the visible image. Next, the proposed formula is applied to process the Cb and Cr components of the original visible image. The reflectance component features and infrared features are concatenated and input into the fusion network to obtain the Y component of the fused image. Finally, the Y component is concatenated with the processed Cb’ and Cr’ components of the visible image to generate the final fused image.

In summary, the main contributions of this paper are as follows:

• This paper proposes a two-stage network for joint low-light image enhancement and image fusion, which can alleviate overexposure and block artifacts while eliminating color deviation, named CDFFusion;
• A brightness enhancement formula without color deviation is proposed, which processes the three components (Y, Cb, Cr) simultaneously, and the processed results have no color deviation;

2. Related Work

Deep learning-based visible and infrared image fusion methods can be roughly divided into three categories: those based on Convolutional Neural Networks (CNN), those based on Autoencoders (AE), and those based on Generative Adversarial Networks (GAN). Zhang et al. proposed a Squeeze-and-Decomposition Network, SD-Net [2], which models image fusion as the extraction and reconstruction of gradient and intensity information. Based on the fact that different fusion tasks share a similar goal—fusing images by integrating important and complementary information from multiple source images—Xu et al. proposed a unified unsupervised end-to-end image fusion network, U2Fusion [3], which is characterized by its ability to perform information measurement on extracted features to automatically estimate the importance of source images. Li et al. proposed an end-to-end fusion network architecture, RFN-Nest [4], which includes an encoder, a Residual Fusion Network (RFN), and a decoder. The encoder extracts multi-scale deep features through max-pooling; the RFN is composed of several convolutional layers and is trained with a new loss function to implement a learnable fusion strategy instead of manually designed rules; the decoder adopts an architecture based on nested connections to reconstruct the fused image. Xu et al. proposed a classification saliency-based pixel-level fusion method, CSF [5], which classifies different source images, uses the classification results to represent the saliency of each pixel in the source images, and finally fuses the feature maps using this saliency to generate the fusion result. Ma et al. first proposed an end-to-end model based on Generative Adversarial Networks (GAN) and named it FusionGAN [6]. It formulates the fusion problem as an adversarial problem while avoiding the manual design of complex activity level measurement and fusion rules. Later, the team designed another Generative Adversarial Network with Multiclassification Constraints, GANMcC [7], which formulates the fusion problem as the simultaneous estimation of multiple distributions.

DIVFusion [8] was the first to combine IVIF and low-light enhancement. It decomposes the visible image into reflectance features and illumination features at the feature level, then enhances and fuses the reflectance features with infrared features. LENFusion [9] adopts the idea of pre-enhancement, re-enhancement, and fusion. It draws on the idea of PIAFusion [10] by pre-training a binary classifier and using its classification results as part of the loss function of the backbone network. LEFuse [11] introduces a hybrid module of Transformer and CNN and designs the overall network structure into a symmetric structure similar to the U-net style.

3. Methods

The proposed method in this paper improves nighttime visibility, retains the complementary information of the two modalities, alleviates overexposure and block artifacts, and eliminates color deviation. This section details the two sub-networks of the entire framework, including the network structure and loss function.

3.1. Overall Framework

All As shown in Figure 3, let $I_{vi}^Y$ and $I_{ir}$ represent the Y channel of the visible image and the infrared image respectively. After passing through the decomposition network and fusion network, the Y channel of the fused image is obtained. Then, it is concatenated with the processed Cb and Cr channels of the visible image to get the final fusion result. The entire process is divided into two stages: feature extraction and image fusion.

3.2. Reflectance-Illumination Decomposition Network(RID-Net)

The specific structure of the reflectance-illumination decomposition network is shown in Figure 4, which consists of two parts: a visible image reconstruction network and an infrared image reconstruction network. The former is used to separate the reflectance and illumination components of the visible image at the feature level and reconstruct them into images, while the latter is used to extract deep features of the infrared image and reconstruct them into images. First, in the feature extraction stage, the Y channel of the visible image is input into the encoder with an FPN structure to extract deep features $\varphi$. The specific structure of the encoder is shown in Figure 5. The original image passes through 4 convolutional layers (C1-C4) in sequence to obtain 4 feature maps of different sizes. The parameters of C1-C4 are listed in Table 1. Then, 4 1×1 convolutional layers (C5-C8) are used to unify the number of channels of the feature maps obtained just now, which is set to 256 here. After that, these processed feature maps are upsampled by 2 times and summed to complete multi-scale fusion. This process can be expressed as:

$$\varphi =E(I_{vi}^Y)$$

(1)

In the feature separation stage, according to the RetiNex theory, an image can be decomposed into the product of the reflectance component (R) and the illumination component (L) (as shown in the following formula). Therefore, two Efficient Channel Attention (ECA) modules are used to separate the reflectance features ${\varphi }_R$ and illumination features ${\varphi }_L$ from them.

$$I=R\times L$$

(2)

The structure of the ECA module is shown in Figure 6. First, global average pooling is performed on each channel of the input feature map to obtain the global feature of each channel. Then, one-dimensional convolution and activation are applied to these global features to obtain attention weights. The input feature map is reweighted using these weights to get the output feature. Here, GAP and $\sigma$ represent global average pooling and sigmoid activation function respectively. This process can be expressed as:

$$\begin{array}{l}EC{A}_R(\varphi )={\varphi }_R\\ EC{A}_L(\varphi )={\varphi }_L\end{array}$$

(3)

In the final decoding stage, they are input into two decoders with identical structures, and the reflectance component $I_R$ and illumination component $I_L$ are obtained through reconstruction. Both decoders are composed of 4 stacked groups of convolutional and activation layers. The first 3 activation layers use the LRelu function, and the last one uses the sigmoid function. This process can be expressed as:

$$\begin{array}{l}{D}_R({\varphi }_R)=I_R\\ D_L({\varphi }_L)=I_L\end{array}$$

(4)

It should be noted that the decoding part here is only for better image reconstruction and does not participate in the operation of the next stage.

Similarly, the infrared reconstruction network adopts a basically consistent structure, which will not be elaborated here.

The loss function of this part of the network is as follows:

$$L_1={\lambda }_1{L}_{rev}+{\lambda }_2{L}_{per}$$

(5)

The loss function $L_1$ of the visible image reconstruction network consists of two parts: reconstruction loss $L_{rev}$ and perceptual loss $L_{per}$. The reconstruction loss $L_{rev}$ is derived from Formula (2), which is expressed as follows:

$$L_{rev}={‖I_R\times I_L-I_{vi}^Y‖}_1$$

(6)

On this basis, we hope that the brightness-enhanced result (i.e., the reflectance component $I_R$) and the result of the original image $I_{vi}^Y$ after histogram equalization have as similar representations as possible in the feature domain of the VGG-19 network. Thus, the perceptual loss $L_{per}$ is defined as:

$$L_{per}={‖VGG(I_R)-VGG[hist(I_{vi}^Y)]‖}_2^2$$

(7)

where “hist” represents the histogram equalization operation. Here, the Conv4-1 feature of the VGG-19 network is selected.

The loss function $L_2$ of the infrared reconstruction network consists of reconstruction loss $L_{rei}$ and structural similarity loss $L_{s1}$, which can reconstruct results similar to the original image in both intensity and structure. It is defined as follows:

$$\begin{array}{l}{L}_2={\lambda }_3{L}_{rei}+{\lambda }_4{L}_{s1}\\ L_{rei}={‖I_{ir}\prime -I_{ir}‖}_1\\ L_{s1}=1-ssim(I_{ir}\prime ,I_{ir})\end{array}$$

(8)

Here, ${\lambda }_1\sim {\lambda }_4$ are weight hyperparameters used to balance the various parts of the loss.

3.3. Fusion Network

In the fusion stage, the reflectance features ${\varphi }_R$ of the visible image and the infrared features ${\varphi }_{ir}$ are concatenated at the channel level and input into the fusion network. Consistent with the decoder structure in the previous stage, it is also composed of 4 stacked groups of convolutional and activation layers. At this time, its output $I_f^Y$ is taken as the Y channel of the fused image, which is used to participate in the final channel fusion.

According to the ITU-R BT.601 international standard, the conversion formula from the RGB space to the YCbCr space of an image is:

$$\begin{array}{l}Y=0.299R+0.587G+0.114B\\ Cb=128-0.168736R-0.331264G+0.5B\\ Cr=128+0.5R-0.418688G-0.081312B\end{array}$$

(9)

In the HSI color space, the conversion formula from RGB to HSI is:

$$\begin{array}{l}H=\left\{\begin{array}{c}\theta ,B\le G\\ {360}^{\circ }-\theta ,{\rm B}>G\end{array}\right.\\ \theta =\arccos \left\{{\displaystyle \frac{\frac{1}{2}[(R-G)+(R-B)]}{[{(R-G)}^2+(R-B){(G-B)}^{1/2}]}}\right\}\\ S=1-{\displaystyle \frac{3}{(R+G+B)}}[\min (R,G,B)]\end{array}$$

(10)

where H represents hue and S represents saturation. From the above formula, it can be proved that: only when the three components R, G, and B are scaled proportionally, the hue and saturation of the pixel remain unchanged. Substituting the result into Formula (9), it can be obtained that: only when (Cb - 0.5) and (Cr - 0.5) are scaled synchronously with Y, R, G, and B are scaled proportionally (the pixel value range has been normalized to [0, 1]), and the hue and saturation remain unchanged. Thus, we naturally derive the mapping formula for the Cb and Cr channels of the visible image:

$$\begin{array}{l}scale=I_R/I_{vi}^Y\\ I_{vi}^{Cb}\prime -0.5=(I_{vi}^{Cb}-0.5)\times scale\\ I_{vi}^{Cr}\prime -0.5=(I_{vi}^{Cr}-0.5)\times scale\end{array}$$

(11)

where “scale” is the brightness gain image. Different from other methods that use loss functions for weak constraints, the above formula applies strong constraints on the proportion between the three components R, G, and B of the visible image, thus avoiding color distortion. Finally, $I_f^Y$ is concatenated with $I_{vi}^{Cb}\prime$ and $I_{vi}^{Cr}\prime$, and converted back to the RGB space to obtain the final fused image $I_f$.

The loss function $L_3$ of the fusion network is as follows:

$$L_3={\lambda }_5{L}_{vi}+{\lambda }_6{L}_{ir}+{\lambda }_7{L}_{aux}+{\lambda }_8{L}_{grad}+{\lambda }_9{L}_{s2}$$

(12)

It consists of visible light intensity loss $L_{vi}$, infrared intensity loss $L_{ir}$, auxiliary intensity loss $L_{aux}$, gradient loss $L_{grad}$, and structural similarity loss $L_{s2}$, which are defined as follows:

$$\begin{array}{l}{L}_{vi}={‖I_f^Y-I_R‖}_1\\ L_{ir}={‖I_f^Y-I_{ir}‖}_1\\ L_{aux}={‖I_f^Y-\max (I_R,I_{ir})‖}_1\\ L_{grad}=‖\nabla I_f^Y-\max (\nabla I_R,\nabla I_{ir})‖\\ L_{s2}=1-{\displaystyle \frac{1}{2}}[ssim(I_f^Y,I_R)+ssim(I_f^Y,I_{ir})]\end{array}$$

(13)

where $\nabla$ represents the gradient operation, and the Sobel operator is used here. These losses can force the fused image to retain more prominent intensity and gradient information from the source images and maintain a relatively consistent structural similarity with the source images. Here, ${\lambda }_5\sim {\lambda }_9$ are weight hyperparameters used to balance the various parts of the loss.

4. Experiments

4.1. Experimental Configuration

To comprehensively evaluate the proposed method, extensive experiments are conducted on the LLVIP dataset [12]. The LLVIP dataset is a paired visible-infrared dataset for low-light scenarios, containing 33,672 images (16,836 pairs). Among them, 240 pairs of infrared and visible images are selected for the training phase, and 50 pairs are selected for the testing phase. These images have been strictly registered. The results of this paper are compared with 5 fusion methods, including 1 AE-based method (RFN-Nest), 1 GAN-based method (FusionGAN), and three latest nighttime IVIF methods (DIVFusion, LEFuse, and LENFusion). The implementation of all methods is based on publicly available code.

In the quantitative evaluation phase, six metrics are used, including 1 image feature-based metric (Spatial Frequency, SF), 1 structural similarity-based metric (Multi-Scale Structural Similarity, MS-SSIM), and 4 metrics based on the source images and generated images (Correlation Coefficient, CC; Sum of Correlation Differences, SCD; Gradient-Based Fusion Performance, Qabf; and Noise-Based Fusion Performance, Nabf). Among them, a smaller Nabf value indicates a better fusion effect.

In the training phase, the 240 pairs of images used for training are randomly cropped to a size of 224×224 pixels, and the batch size is set to 5. The initial learning rates of the visible and infrared images in the decomposition network are 10^-4 and 10^-3 respectively, and they are decayed to 0.1 and 0.01 of the initial values after 50 and 75 epochs. The fusion network uses a fixed learning rate set to 2×10^-5. In addition, the weight hyperparameters of the loss functions of the decomposition network and fusion network are set as follows:

$${\lambda }_1=10, {\lambda }_2=5, {\lambda }_3=10, {\lambda }_4=1, {\lambda }_5=2, {\lambda }_6=2,{\lambda }_7=6, {\lambda }_8=10, {\lambda }_9=30$$

(14)

4.2. Results and Analysis

4.2.1. Qualitative Comparison

Figure 7 shows the visualization results of different fusion methods. As mentioned earlier, previous fusion methods fail to extract the information of visible images obscured at night. In the results of RFN-Nest and FusionGAN, the upper part of the images in the first row is still completely black, with no significant improvement in brightness, and the zebra crossings in the red boxes become blurred, with the zebra crossings in the upper right corner showing a greenish tint. In the second row of images, most details of the road in the boxes are lost, and the headlight area in the lower right corner is covered with a shadow. In the third row of images, the road details are completely lost. For the results of DIVFusion, LEFuse, and LENFusion, the color saturation in figures (e) and (f) is relatively low, leading to an overexposed feeling in some areas (as shown in the green boxes in the second row of images). In figure (g), areas such as zebra crossings are prone to distortion, resulting in information loss (as shown in the red boxes in the first and second rows of images). In areas such as damaged roads, the color is darkened, forming unnatural false edges (as shown in the green box in the first row of images). In addition, these three methods all cause block artifacts (as shown in the boxes in the third row of images). In the third row of images, the color saturation of buildings and leaves is significantly low, with varying degrees of color deviation. In contrast, the proposed method alleviates all the above issues and achieves a better visual effect without color deviation.

4.2.2. Quantitative Comparison

Table 2 shows the average quality metrics of different fusion methods on the LLVIP dataset. It can be seen that the proposed method ranks first in the Qabf and MS-SSIM metrics, second in the CC and SCD metrics, and third in the SF and Nabf metrics. This indicates that the proposed method can suppress block artifacts while maintaining good high-frequency details, and can well maintain the consistency between the fused image and the source images.

4.3. Generalization Experiment

To verify the generalization ability of the proposed model, 48 pairs of images are selected from the M3FD dataset [13] for testing. The model is not trained on the M3FD dataset and is directly used for testing. The M3FD dataset is a visible-infrared fusion dataset, containing 4,200 pairs of images for fusion, detection, and fusion-based detection, as well as 300 pairs of images for independent scene fusion.

4.3.1. Qualitative Comparison

As shown in Figure 8, first, in the result of RFN-Nest, the information on the signs is obscured in darkness and cannot be seen clearly (as shown in the green box in the first row of images and the red box in the second row of images). Secondly, the results of RFN-Nest and FusionGAN are generally blurrier than the original images (such as the tree areas in the first row of images). In the results of DIVFusion, LEFuse, and LENFusion, snowflake-like noise appears in the sky area of the first row of images, and the outline of the clouds is eroded (as shown in the red box in the first row of images). In the second row of images, the outline of distant buildings becomes incomplete compared with the original image. In addition, the results of these three methods all have varying degrees of color distortion, and figure (f) even gives an overexposed feeling.

4.3.2. Quantitative Analysis

Table 3 shows the average quality metrics of different fusion methods on the M3FD dataset. It can be seen that the proposed method ranks first in the Qabf metric, second in the SF and CC metrics, and third in the Nabf, SCD, and MS-SSIM metrics. This indicates that the proposed method can retain rich texture details and well maintain the correlation and consistency between the fused image and the source images.

4.4. Ablation Experiment

To verify the effectiveness of each loss function in the proposed method, an ablation experiment is designed. To verify the effectiveness of the perceptual loss $L_{per}$ and auxiliary intensity loss $L_{aux}$, these two loss functions are removed respectively, and the results are shown in Figure 9.

4.4.1. Qualitative Comparison

When the perceptual loss $L_{per}$ is removed, the fused image exhibits severe color deviation and block artifacts. In fact, at this time, the RID-Net cannot accurately separate the reflectance component $I_R$. When the auxiliary intensity loss $L_{aux}$ is removed, the fused image fails to express sufficient infrared information (as shown in the green box in Figure (d)), and a lot of road texture information is lost (as shown in the red box in Figure (d)).

4.4.2. Quantitative Analysis

The quantitative results of the ablation experiment are shown in Table 4. It can be seen that the proposed method achieves better results in the CC, Qabf, and SCD metrics. The SF is not the optimal because areas with block artifacts often have higher spatial frequencies. The MS-SSIM is also not the optimal because without the auxiliary intensity loss $L_{aux}$, the fused image tends to be closer to either the visible image or the infrared image.

5. Conclusions and Discussion

This paper proposes a new fusion method to address the shortcomings of existing visible and infrared image fusion methods in low-light scenarios. CDFFusion adopts a two-stage strategy to solve the problems of overexposure and block artifacts, and completely avoids color deviation. Specifically, we first train a feature extraction network that can extract features of source images and separate the illumination component and reflectance component of the visible image. Then, a fusion network is trained, which combines the proposed color mapping formula to realize fusion of infrared images and enhanced visible images without color deviation. Experimental results show that compared with the existing five methods, the proposed method achieves generally better results in both subjective and objective aspects.

The limitations of this paper are as follows: the encoder with the FPN structure has certain requirements on the size of the input image, requiring both the height and width of the image to be integer multiples of 8. Images that do not meet the size requirements need to be preprocessed. In addition, there are certain shortcomings in the feature extraction of infrared images, and the unique advantages of infrared images are not separated and fully utilized.