R-PreNet: Deraining Network Based on Image Background Prior

1. Introduction

Rain is a common weather phenomenon which causes adverse effects on the visual quality of images and affecting the performance of subsequent image processing tasks, such as object recognition [1], object detection [2], autonomous driving and video surveillance [3,4,5], etc. Hence, removing rain streaks from images with rain has become an important and meaningful research topic, and has also received attention in recent years [6,7,8]. Single image deraining refers to restoring a clean and rainless image scene from a single image with rain. However, due to the complex combination of background information and raindrop information, how to simultaneously remove raindrops and protect the background remains a challenging issue. We found in the experiment that the PreNet deraining network model [6] can reconstruct a relatively clear rain free image, but in the test of rainstorm data set, the background structure of the reconstructed image corresponding to the rainstorm image has also been damaged to some extent, that is, the introduction of artifacts, and this destruction of the image background will sometimes lead to serious problems, such as fuzzy or missing traffic signs may lead to serious accidents in automatic driving. In order to address this problem, this paper introduce an additional image background prior to protect the background structure, so that a clearer and correct reconstruction of rainless images can be obtained in the case of rainstorm, as shown in Figure 1.

In this article, we explore the effective reconstruction problem of complex combinations of background and raindrops, and propose a new algorithm called R-PReNet that can effectively remove raindrops and protect background information. This algorithm fully utilizes the cyclic recursive structure and raindrop energy of PReNet. In addition, this article introduces residual channel prior (RCP) [9,10,11,12] in the model to achieve background structure protection. In addition, this article also proposes the use of the 'Squeeze Excitation' residual module (SE ResBlock) [13] to extract deep features of RCP, and the interactive fusion feature module (IFM) [12] to fully utilize RCP information, achieving high-quality rainless image reconstruction.

Our contributions are summarized as follows:

This article replicates and tests the PreNet deraining network on three popular image deraining datasets (Rain100H [15], Rain100L [15], Rain14000 [16]) and real datasets (Practical_by_Yang [15]), and studies the results of deraining.

This article explores the effectiveness of residual channel prior (RCP) for background protection and proposes an image deraining network structure based on RCP. Numerous experiments have shown that our method outperforms the original method on commonly used rainfall datasets, restoring visually clean images and good details.

We propose an RCP extraction module and an interactive fusion module (IFM) for RCP extraction and guidance, respectively, to obtain deep features of RCP and guide the network to recover more background details.

The rest of this article is as follows. In Section 2, we briefly reviewed the relevant research on image deraining methods. In Section 3, we proposed an overall R-PreNet deraining network based on image background prior, and elaborated on the RCP residual channel prior and IFM fusion methods in detail. In the fourth section, we presented our experimental results and comparisons. The conclusion is given in Section 5.

2. Related Works

The goal of the single image deraining task is to restore rainless images from images with rain. The video based deraining task can use video to obtain consecutive multiple frames of images, and use the temporal nature of the continuous images to obtain the position information of rain streaks and the background information of rain streaks occlusion position, thus achieving the video based deraining task. However, the task of removing rain from a single image lacks the assistance of the location information of rain streaks and the background information of occlusion positions, making it difficult to reconstruct a single rainless image.

The existing methods proposed for this task can be mainly divided into two categories: model driven methods and data driven methods.

2.1. Model-driven Methods

Generally speaking, early filter based methods and traditional prior based methods belong to the model driven type. We will introduce representative works of these two aspects as follows.

An image can be decomposed into low-frequency and high-frequency parts, with details and noise information mainly distributed in the high-frequency part of the image. So, it can be seen that the raindrops in the rain image are mainly distributed in the high-frequency part. Therefore, in the early stage of removing rain from a single image, a guided filter [17] is introduced as a general tool for image prior representation, decomposing the rain image into low-frequency part (LFP) and high-frequency part (HFP). Subsequently, Xu et al. [18], Zheng et al. [19], Ding et al. [20], Kim et al. [21] utilized the characteristics of rain streaks and different guided filtering methods to remove rain from a single rain image, achieving preliminary success. However, there are still issues such as leaving obvious rain streaks and missing background details, so there is room for further performance improvement in this method.

Broadly speaking, we consider an image with rain to be composed of clear background layers and rain layers:

$$\mathrm{O}=\mathrm{B}+\mathrm{S}$$

(1)

B represents the background layer, which is the target image to be obtained; S represents the rainmark layer; O represents the input image with rain marks, so the problem of removing rain can be expressed as an image decomposition problem based on dictionary learning and sparse representation. Therefore, later scholars no longer only rely on different guided filtering methods to remove rain from a single rain image, but began to study the physical properties of rain streaks themselves (such as sparsity and Self-similarity), and introduced them into the deraining model as prior information, thus realizing the reconstruction of rainless images.

Kang et al. [22,23] first decomposed the image into high-frequency and low-frequency components using Bilateral filter, and then decomposed the high-frequency components into "rain component" and "no rain component" through dictionary learning and sparse coding, deleting the rain component from the image, and retaining most of the original image details. This algorithm focuses on training in the high frequency layer rather than the image domain, so it has the advantages of reducing computing resources and no interference in the low frequency layer. However, this method is very time-consuming, and because it relies too much on the preprocessing of Bilateral filter, the background is usually fuzzy, so there is still room for further optimization of performance.

In order to further obtain a clear background layer, we comprehensively explored more inherent properties of the background and raindrops, and regularized them to constrain the solution space. The classic methods include (1) low rank appearance induced by non local similarity of raindrops [24]; (2) Gaussian Mixture model (GMM) to calculate the distribution of rain streaks at different scales and directions [25,27]; (3) A sparse representation model based on some learning rain atoms [26]. Although these methods have modeled and referenced both rain streaks and background layers, they can only handle light rain streaks, making it difficult to handle heavy or sudden rain streaks, and there is still a problem of time-consuming processing.

We have studied these model driven methods and found that although incorporating physical prior information about rain patterns and background layers can help achieve rain image reconstruction, these prior information is usually subjective and incomplete, making it difficult to fully transfer existing prior knowledge in real rain images [27,28,29]. Especially, the rain maps obtained from real scenes are often complex and variable, so the performance of directly establishing models for removing rain is always unsatisfactory. Therefore, data-driven deep learning deraining algorithms have become the latest trend in deraining tasks.

2.2. Data-driven Methods

With the development of deep learning theory and technology, data-driven single image deraining methods are becoming increasingly popular. These methods automatically extract features from the dataset through network structures, thereby achieving mapping from rain images to deraining images.

Since 2013, Eigen et al. [30] trained a special CNN by minimizing the mean square deviation between predicted rain and no rain image blocks, and for the first time used deep learning methods to remove raindrops attached to images. To this day, many CNN based image denoising deep networks have been proposed [16,28,31]. Usually, in these deep neural networks, constraints related to rainfall, such as rainfall masks and background features, are added to the network to learn features more comprehensively. Later, some methods utilized cyclic networks and residual networks [6] to gradually remove raindrops, which streamlined the network structure and reduced network parameters.

However, due to the difficulty in obtaining paired real rain maps and rainless images, there are differences between synthetic rain maps and real rain maps. Previous deraining algorithms may result in certain performance deviations when directly applied to real rain maps. To address the above issues, experts have considered introducing unsupervised and semi supervised methods in image deraining networks.

Semi-Supervised Learning is to use both unlabeled data and labeled data for learning. For example, Wei et al. [27] proposed SIRR. The network simulates the real rain residue through the likelihood term applied to the Gaussian Mixture model, and minimizes the KullbackLerbler divergence between the synthetic rain and the real rain distribution; Yasala et al. [32] proposed Syn2real (GP), which uses Gaussian process to model the potential features of rain images and create false labels for unlabeled data; Huang et al. [8] proposed MOSS, which uses a memory oriented decoder encoder network to learn rainfall patterns and restore rainless background images. These methods improve the generalization ability of the deraining algorithm by jointly mining the features of rain streaks from real and synthetic data using different methods.

Unsupervised learning means that it does not rely on the marked data and directly models the input data. The unsupervised algorithm in the deraining algorithm is implemented by introducing the Generative adversarial network (GAN) Zhu et al. [34] proposed an unsupervised end-to-end adversarial deraining network, RainRemoval GAN (RR GAN), which can generate real rain free images using only unpaired images. The network mainly includes a multi-scale attention memory generator and a multi-scale attention discriminator, and its architecture is still similar to the supervised GAN method; Jin [33] proposed another unsupervised Generative adversarial network (UD-GAN), which introduced self supervision constraints into the internal statistical information of unpaired rain and clean images. It uses two mutually cooperative modules, namely, the background guidance module (BGM) and the rain guidance module (RGM), where RGM is used to distinguish between real rain free maps and fake rain free maps generated based on BGM, BGM uses a hierarchical Gaussian blur gradient error to ensure the background consistency between the rain map input and the deraining output.

Afterwards, there were also unsupervised algorithms such as DerainCycleGAN [7] which greatly solved the problems of difficulty in obtaining paired real rain images and rainless images, as well as poor generalization ability of algorithms based on synthetic images. However, there are still problems such as overly complex networks, time-consuming training, and insufficient rain pattern removal.

Although these data-driven methods can effectively remove some rain streaks, they cannot remove all rain streaks in complex situations such as rainstorm. At the same time, these methods are also difficult to fully protect the structural information of the image, and even introduce new artifacts in the reconstructed image. Therefore, a method that is simple and efficient, removes a large number of raindrops, protects object structures, and improves generalization ability is crucial.

3. Proposed Work

In this section, we introduce the overall network architecture of the algorithms in this paper. We first describe the proposed residue channel prior (RCP) implementation details. Then We show the structure of progressive recurrent network (PReNet) as a backbone network. Finally, we propose methods for fusing high-dimensional features of the RCP.

3.1. Residue-Progressive Recurrent Network

As shown in Figure 2, R-PReNet consists of two main parts: (i) the RCP feature extraction and fusion module, and (ii) the progressive recurrent network. We first extract and fuse image features and RCP features from the rainy image. Then we concatenate the fused features with image features. The components of our method are described in detail in the following sections.

3.2. Residue Channel Prior (RCP)

The occurrence of rain streaks is usually modeled as a linear combination of a background layer and a rain layer [16,24,36,37]. Based on this model, Li et al [9] demonstrated that subtracting the minimum color channel from the maximum color channel will generate a rain-free image. Rain streaks are colorless (white or grey) and appear at the same location in different RGB color channels. So subtracting the minimum color channel from the maximum color channel will cancel the appearance of rain streaks as in Figure 3.

The colored-image intensity of a rainy image is defined [9] as:

$$\tilde{I}\left(x\right)=\tau {\rho }_{rs}\left(x\right)L\sigma +\left(T-\tau \right)B\pi$$

(2)

where L $={(L_r,L_g,L_b)}^T$ is the color vector of luminance and B $={(B_r,B_g,B_b)}^T$ is the color vector of background reflection.

$$L=L_r+L_g+L_b,B=B_r+B_g+B_b$$

(3)

In the model (Eq. (2)), the first term is the rain streak term and the second term is the background term. $\mathsf{\sigma }=\mathbf{L}/L$ and $\mathsf{\pi }=\mathbf{B}/B$ are defined the chromaticities of L and B. T is the exposure time and τ is the time for the raindrop to pass through pixel x. ${\rho }_{rs}$ consists of the refraction coefficients of the raindrop, the specular reflection coefficients, and the internal reflection coefficients. We assume that ${\rho }_{rs}$ is wavelength independent, which implies that the raindrop is colorless.

As a consequence, we need to cancel the light chromaticity σ in the rain-streak term in Eq. (2) to generate a residual channel without rain streaks. To do so, we use any existing color constancy algorithm [38] to estimate σ, and then apply the following normalization step to the input image.

$$\mathrm{I}\left(\mathrm{x}\right)=\frac{\tilde{I}\left(x\right)}{\sigma }=I_{rs}\left(x\right)i+I_{bg}\left(x\right)$$

(4)

where, $\mathrm{i}={(1,1,1)}^T$, $I_{rs}=\mathsf{\tau }{\rho }_{rs}L$, $I_{bg}=(\mathrm{T}-\mathsf{\tau })B/\sigma$.

Vector division is done element-wise. Note that when we normalize the image, we cancel not only the light chromaticity but also the color effect of spectral sensitivity.Hence, according to the previous equation and a rainy image I, the residual channel is defined as:

$$I_{res}\left(x\right)=I^M\left(x\right)-I^m\left(x\right)$$

(5)

where,

$$I^M\left(x\right)=\max \{I_r\left(x\right),I_g\left(x\right),I_b\left(x\right)\}$$

(6)

$$I^m\left(x\right)=\min \{I_r\left(x\right),I_g\left(x\right),I_b\left(x\right)\}$$

(7)

$I_{res}$ is the residual channel of the image I, which has no rain streaks.

3.3. RCP High-Dimensional Feature Extraction

Although the operation of subtracting a color channel from another color channel in the image space is useful and the structural information of the RCP is clearer than the rainy image, it can be destructive to the background image because of information loss. Therefore, we move the operation that utilizes the RCP structural information to the feature domain. We propose the RCP feature extraction module to extract the high dimensional features of RCP.

Based on the Squeeze-and-Excitation (SE) block proposed by Hu et al [14], which focuses on channel relationships to construct informative features, this residual block adaptively recalibrates the channel feature responses by explicitly modeling the interdependencies between channels. Since the RCP module is a module that uses color channels to interact with each other, in order to reduce the noise in the initial features and enrich the semantic information of the features, we use the SE-ResBlock structure shown in Figure 4 to extract the high dimensional features Fp of the RCP.

3.4. Interactive Fusion Features

We extract the high dimensional features of RCP, but it is still a challenging task that fully utilize the RCP features to guide the model.

A simple solution is directly concatenating RCP features with image features, but this is ineffective for guiding model deraining and may cause feature interference. To address this problem, we propose an interactive fusion module (IFM) [39] consisting of two branches (rainy image features and prior features) to progressively combine features. As shown in Figure 5, two 3 × 3 kernel-sized convolutions are performed to map the rainy image features F_o and RCP features F_p to $\widehat{F_o}$and $\widehat{F_p}$.

Next, the similarity map S between $\widehat{F_o}$ and $\widehat{F_p}$ is computed using element multiplication:

$$\mathrm{S}=\mathrm{S}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}(\widehat{F_o}\otimes \widehat{F_p})$$

(8)

The background information of the rainy image corrupted by the rain streaks is enhanced using the similarity map S. In addition, since the background of the RCP is similar to the rainy image, the similarity map S can also highlight the feature information in the priori features, which further enhances the structure of the priori features.

3.5. Progressive Recurrent Network

The progressive recurrent network consists of the following four parts: (i) a convolutional layer $f_{in}$ receives network inputs, (ii) a recurrent layer $f_{recurrent}$propagates cross-stage feature dependencies, (iii) several residual blocks $f_{res}$ extracts the deep representation, and (ii) a convolutional layer $f_{out}$ outputs deconvolutional results. Where $f_{in}$ takes as input the current estimation ${\mathrm{x}}^{t-1}$, the rainy image y, and the concatenation of the background fusion prior features G. The recurrent layer we implement using convolutional Long Short-Term Memory (LSTM) because LSTM has experience advantages in image deraining, through which cross-stage feature dependencies can be propagated to facilitate rain streaks removal.

$${\mathrm{x}}^{t-0.5}=f_{in}({\mathrm{x}}^{t-1},\mathrm{y},\mathrm{G})$$

(9)

$${\mathrm{s}}^t=f_{recurrent}({\mathrm{s}}^{t-1},{\mathrm{x}}^{t-0.5})$$

(10)

$${\mathrm{x}}^t=f_{out}\left(f_{res}\right({\mathrm{s}}^t\left)\right)$$

(11)

where $f_{in}$, $f_{res}$ and $f_{out}$ are stage-invariant, the network parameters are reused in different stages. The recurrent layer $f_{recurrent}$ takes ${\mathrm{x}}^{t-0.5}$ and the recurrent state ${\mathrm{s}}^{t-1}$ as inputs to stage t-1. By unfolding PreNet [6] with T recurrent stages, the deep representation of rain streak removal is favored by recurrent state propagation. The rain removal results from the intermediate stages of the network structure show that the accumulation of storm streaks can be gradually eliminated.

3.6. Loss Function

We employ negative SSIM loss [40] as our objective function. For a model with T stages, we have T outputs, ${\mathrm{x}}^1$, ${\mathrm{x}}^2$, …, ${\mathrm{x}}^T$, and we apply supervision only to the final output ${\mathrm{x}}^T$. The negative SSIM loss is:

$$L=-\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}(x^T,x^{gt})$$

(12)

where$x^{gt}$ is the corresponding ground-truth clean image.

4. Experiments

Our model was trained on Ubuntu OS, NVIDIA GeForce GTX 3080Ti GPU using Pytorch framework in Python environment with 12GB of RAM. To validate the effectiveness of our model, we evaluated our method on three popular image-deraining synthetic datasets (Rain100H, Rain100L, Rain14000) and a real rainy images dataset (Practical_by_Yang) to evaluate our approach:

Figure 6. Image deraining results tested in both synthetic and real datasets. The first column is the rainy image, the second column is the real rain-free image on the synthetic dataset (no example images on the real dataset), the third column is the deraining result of the PReNet algorithm, and the fourth column is the deraining result of the R-PReNet algorithm of this paper. It can be seen that R-PReNet can reconstruct the rain-free image with clearer background structure and reduce the introduction of artifacts.

Figure 6. Image deraining results tested in both synthetic and real datasets. The first column is the rainy image, the second column is the real rain-free image on the synthetic dataset (no example images on the real dataset), the third column is the deraining result of the PReNet algorithm, and the fourth column is the deraining result of the R-PReNet algorithm of this paper. It can be seen that R-PReNet can reconstruct the rain-free image with clearer background structure and reduce the introduction of artifacts.

4.1. Experimental Setup

4.1.1. A Datasets

In this paper, we mainly use synthetic datasets and real datasets for evaluation. The synthetic image datasets include (1) Rain100L, where 200 pairs of images are used for training and 100 pairs of images are used for testing; (2) Rain100H has 200 synthetic images used for training and 100 images used for testing; and (3) Rain14000, which composed of training and test images with a ratio of 12600:1400 split. The real dataset consists of (1) the Practical_by_Yang dataset with 147 images without ground-truth; and (2) real rainy images from certain movie and television productions.

4.1.2. B Evaluation Indicators

In our experiments, for images with ground truth, we can evaluate each method by two commonly used quantitative metrics, the peak signal-to-noise ratio (PSNR) [41] and the structural similarity index (SSIM) [40]. For the images without ground truth (i.e., real dataset), we provide some visual results.

4.2. Ablation Study

4.2.1. A Effectiveness on RCP module

The first ablation study evaluates the performance of R-PReNet with experimental results with and without the RCP module, we train and test the networks with and without the RCP module and the baseline algorithms of JORDER [28] and RESCAN [31] on the datasets Rain100L, Rain100H, and Rain14000, respectively, and Table 1 shows the performance of the above algorithms on the quantitative results in PSNR and SSIM. Both quantitative and visual results show that the recurrent network with RCP module outperforms the network without RCP module and the baseline algorithm.

4.2.2. B Effectiveness on IFM module

To investigate the effectiveness of the feature fusion module, we compare two different network architectures: (a) with the RCP module, but the RCP high-dimensional features are directly connected with the rainy image features into the network, and (b) with the RCP module and the IFM module, which uses interactive fusion to combine the RCP high-dimensional features and the rainy image features together into the network. We train and test the networks with and without the IFM module and the baseline algorithms of JORDER [28], RESCAN [31], and PreNet [6] on the datasets Rain100L, Rain100H, and Rain14000, respectively, and Table 2 shows the quantitative results of the above algorithms in PSNR and SSIM. Both quantitative and visual results are that the recurrent network with IFM module outperforms the network without IFM module and the baseline network.

5. Conclusion

In this paper, we propose a progressive recurrent deraining network based on background protection. The experiments show that this algorithm can remove rain streaks and protect background information at the same time. In the pre-processing stage of the rainy image, we first extract the residual channel from the rainy image, and the extracted residual channel does not contain rain streaks, while the residual channel is used to extract high-dimensional features, and then we interactively fuse the extracted features with the rainy image features and then input into the progressive recurrent network. The input to the network of each generation is composed of the fused features, the reconstructed image of the previous generation, and the original rainy image. After generations of progressive recursion, the final rain-free image is produced. Comprehensive experimental evaluations show that our method outperforms the original algorithm on both synthetic and real rainy images.