Extended Guided Filter: Simultaneous Noise Suppression and Details Enhancement

1. Introduction

purpose of filtering is to emphasize informative details of interest in the data. Image filtering is widely used in the field of image processing and computer vision for a broad spectrum of applications, such as image denoising[1,2,3,4], smoothing [5,6,7], image fusion [8] and feature extraction [9,10,11], etc. Observed from the frequency domain perspective, spatially invariant filters, e.g. Gaussian averaging filter, give a modulation scheme to the frequency spectrum of the input image, allowing the energy of specific frequencies to pass while suppressing that of certain frequencies. This leads to a global tradeoff between the signal energy and the noise energy. However, meaningful details and noise of natural images always share certain frequencies. As an result, unfortunately, those linear and spatially invariant filters often inevitably damage the meaningful details of an image to a certain extent.

To alleviate this, nonlinear methods such as anisotropic diffusion[5,12,13,14,15], bilateral filter[4,16,17,18,19], WLS filter[20,21] and guided image filter (GIF)[3,22,23,24] have been proposed. They take local content into consideration and can effectively prevent edge pixels from being damaged in denoising. Featuring the advantages of both simple framework and high performance, GIF has established itself as an exemplar for edge-preserving filtering. When the guiding image is the input itself, GIF implicitly measures local noise level based on variance, and thus determines the local combination coefficients between the original image and a constant valued image accordingly.

However, there are two issues of GIF that deserve further improvement. First, we noted that variance, a second-order statistic employed in GIF, is limited in its ability to discriminate between noise and details, and tends to cause tiny structures to be suppressed along with noise. Specifically, the variance of an area with meaningful details in an image is often indistinguishable from the variance of noise (see Figure 2), which is why the edge-preserving filters such as GIF can only preserve significant edges but lose many meaningful details. Second, the focus of GIF and those edge-preserving filters [3,4,14] is on how to smooth the image without destroying the edges rather than enhancing details [25]. Actually, details enhancement is considered as one of the important goals of image filtering besides denoising [26]. Meaningful details carry rich information and should be further emphasized in filtering for the purpose of achieving better image contrast.

In this paper, we propose an extended guided filter (EGF), see Figure 1, primarily motivated by two considerations: Firstly, we regard image filtering as a spectral energy reconfiguration problem, and adaptively regulate the gain of each spectral band to achieve simultaneous noise suppression and detail enhancement locally. Secondly, instead of using variance, we employ a linear noise estimator in the closed-form optimization, to obtain the online SNR measurement required for the dynamical regulation of the signal energy and the noise energy.

2. The Methodology

2.1. The definition of the extended guided filter (EGF)

We assume that filtering is a local recombination of the energy at different spectrum bands of the input image. Therefore, image filter can be regarded as a local linear model, which transform the maps, provided by performing spectrum-selectively filtering on the input, to the output.

A set of spatial filters $b_1$,..., $b_n$,$b_{n+1}$ are given targeting different spectral bands of the input image p. A n+1 dimensional functional space is spanned by those filter bases, ${\mathbf{S}}^{n+1}=\mathrm{span}(b_1,...,b_n,b_{n+1}).$ Each point of the space, ${\sum }_{k=1}^{n+1}{a}_k{b}_k$, represents a combined filter. Tensor $\mathbf{u}$ is yielded to contain a series of maps $u_1$,...,$u_n$,$u_{n+1}$ from p, by performing convolution $u_k=p\otimes b_k.$ The task of EGF is to seek locally in the space for such combined filters that meets certain constraints by means of optimization. The output of EGF is denoted as g. locally, $g^l$, the output of an area ${\Omega }_l$ centered at pixel l, is defined as a linear combination of those $n+1$ corresponding patches $u_k^l$,The guided image filter is an effective method for balancing noise and preserving details based on local variance, achieving noise suppression while retaining prominent edges. However, we have observed limitations in using variance alone to distinguish between noise and image details, resulting in the suppression of fine structures alongside noise. Furthermore, the goal of enhancing image details, which is crucial in image filtering, has not been adequately addressed in edge-preserving filter design. In this study, we approach image filtering as a problem of spectral energy reconfiguration, where the gain of each spectral band is adaptively regulated to achieve simultaneous noise suppression and detail enhancement. Instead of relying on variance, we employ a linear noise estimator to obtain the Signal-to-Noise Ratio (SNR) measurement necessary for dynamic gain regulation. Similar to the guided image filter, we determine the filter coefficients by solving the closed-form solution of the optimizer. Our experiments demonstrate that the proposed extended guided filter effectively integrates information from multiple spectral bands at a local level, allowing for the joint enhancement of meaningful details while simultaneously smoothing noise. $g^l=\left({\sum }_{k=1}^{n+1}{a}_k^l{b}_k^l\right)\otimes p^l={\sum }_{k=1}^{n+1}{a}_k^l{u}_k^l.$ Without losing the generality, we set $b^{n+1}=1$. The coefficients ${\mathbf{a}}^l$ are solved by optimization under the following constrains.

$$E\left({\mathbf{a}}^l\right)=\sum _{{\Omega }_l}({(p^l-g^l)}^2+\sum _{k=1}^n\left({\alpha }_k{\left(a_k^l\right)}^2\right)+\gamma \sum _{{\Omega }_l}{\sigma }^2\left(g^l\right).$$

(1)

The first term of (1) is set to control the error between output and the input. The second term is the coefficient regularizer with hyperparameters $\mathbf{\alpha }$. Besides, the online noise estimation is introduced as the last term with hyperparameter $\gamma$ in (1), to regulate the energy allocation of the signal and the noise. $\sigma$ is a linear estimator of noise level, and thus ${\sum }_{{\Omega }_l}{\sigma }^2\left(g^l\right)={\sum }_{{\Omega }_l}{\left({\sum }_{k=1}^n{a}_k^l\sigma \left(u_k^l\right)\right)}^2.$ Please refer to Section 2.3 for details.

${argmin}_{{\mathbf{a}}^l}\mathrm{E}\left({\mathbf{a}}^l\right)$ could be solved by finding the solution to the linear equations defined by $\frac{\partial E}{\partial a_k^l}=0$, and

$$\left\{\begin{array}{c}{\mathbf{a}}_{1:n}^l=[\mathbf{diag}\left(\mathbf{\alpha }\right)+\mathbf{C}\left({\mathbf{u}}^l\right)+\gamma \mathbf{G}{\left(\sigma \left({\mathbf{u}}^l\right)\right]}^{-1}\mathbf{Cov}(p^l,{\mathbf{u}}^l)\hfill \\ a_{n+1}^l=\sum _l\frac{p}{\left|{\Omega }_l\right|}-\sum _{k=1}^n\left(a_k^l\sum _l\frac{u_k^l}{\left|{\Omega }_l\right|}\right).\hfill \end{array}\right.$$

(2)

Where $\mathbf{Cov}(p^l,{\mathbf{u}}^l)={(\mathrm{Cov}(p^l,u_1^l),\cdots ,\mathrm{Cov}(p^l,u_n^l))}^{\mathrm{T}},$ $\mathbf{C}\left({\mathbf{u}}^l\right)$ is the covariance matrix of ${\mathbf{u}}^l$

$$\mathbf{C}\left({\mathbf{u}}^l\right)=\left[\begin{array}{ccc}\mathrm{Cov}(u_1^l,u_1^l)& \cdots & \mathrm{Cov}(u_1^l,u_n^l)\\ ⋮& \ddots & ⋮\\ \mathrm{Cov}(u_n^l,u_1^l)& \cdots & \mathrm{Cov}(u_n^l,u_n^l)\end{array}\right],$$

(3)

and $\mathbf{G}\left(\sigma \left({\mathbf{u}}^l\right)\right)$ is the Gramian matrix of $\sigma \left({\mathbf{u}}^l\right)$

$$\mathbf{G}\left(\sigma \left({\mathbf{u}}^l\right)\right)=\left[\begin{array}{ccc}\langle \sigma \left(u_1^l\right),\sigma \left(u_1^l\right)\rangle & \cdots & \langle \sigma \left(u_1^l\right),\sigma \left(u_n^l\right)\rangle \\ ⋮& \ddots & ⋮\\ \langle \sigma \left(u_n^l\right),\sigma \left(u_1^l\right)\rangle & \cdots & \langle \sigma \left(u_n^l\right),\sigma \left(u_n^l\right)\rangle \end{array}\right].$$

(4)

For the entire image, if it is processed in a non-overlapping manner, artifacts will appear at each patch boundary. To avoid this, we adopt the solution in [3], where patch centered at each pixel are reconstruct overlappingly with an interval of one pixel. As a result, each location l, will be covered by $\left|{\Omega }_l\right|$ windows and therefore correspond to $\left|{\Omega }_l\right|$ sets of reconstruction coefficients. Here, their mean values are directly used as the reconstruction coefficients of the current pixel. The mean value $\mu \left(a_k\right)$ of $a_k$ is defined as $\mu \left(a_k\right)=BF\left(a_k\right),$ where $BF(.)$ is the mean filter with O(n) time complexity proposed by[3]. The EGF filtering result g is given by $g={\sum }_{k=1}^{n+1}\mu \left(a_k\right).*u_k.$

2.2. Selection of filter bases

In order to construct the functional space ${\mathbf{S}}^{\mathbf{n}+\mathbf{1}}$, filters are selected as the bases of the space. The following factors are considered in the selection process: 1) in order to allow multi-band information integration during filtering, the bases should contain both the low-pass filters and the high-pass filters 2) filters with relatively simple forms are favorable; 3) both the full spectrum pass filter and the DC component pass filter should be included; 4) filters in consideration are expected to be diverse so that they can be complementary.

Filters can either be defined in spatial domain or equivalently be defined in frequency domain. For the sake of simplicity, we choose filters $\left\{B_i\right\}$ with Gaussian form as the space bases in this paper. The spatial domain form $b_i$ of $B_i$ is gvien by Fourier Transform. Among these Gaussian bases, we suppose that $B_1$ is a full-band pass filter and $B_{n+1}$ is a DC component pass filter, respectively. Accordingly, $u_1=p\otimes b_1$ equals the original image p, and $u_{n+1}$ is a constant valued image. The rest of the bases are band-pass filters.

With the bases, the weights for the linear combination for each local area of the image are obtained according to (2). Thus, the linear combination of these bases yields complicated modulation filters in frequency domain. Interestingly, when $n=1$, the form of the combined filter is $a_1^l*p+a_2^l$, which happens to be the linear model adopted by GIF [3].

In this paper, a low-pass filter and a high-pass filter with empirical parameters are chosen as two bases in addition to the full-pass and DC filters that employed in [3]. It is worth noting that, filters collected from Gabor filter bank [27,28] are also encouranged to employe as bases. One can even learn a set of filter bases in a data-driven way.

2.3. Linear noise estimation for dynamic filtering

In order to regulate the power of signal and noise adaptively and optimally, it is necessary to measure the noise level online at any position of the input. Assuming a noise estimator $\sigma$, it gives the results $\sigma \left(x^l\right)$ for a patch $x^l$ of a local region ${\Omega }_l$ centered at the pixel l. Obviously, such an estimator is expected to have the same properties at any image locations. Therefore, the simplest form of such a noise estimator must be a linear & spatially invariant (LSI) system, with the kernel h, and the system output is $h\otimes x^l$. The final noise estimation is given by $\lambda {\sum }_{{\Omega }_l}{\sigma }^2\left(x^l\right)=\lambda {\parallel h\otimes x^l\parallel }_F^2,$ here ${\parallel .\parallel }_F^2$ is Frobenius-norm, and $\lambda$ denote the normalization coefficient.

Interestingly, the authors of [29] have derived a noise estimator kernel

$$h=\left(1,-2,1;-2,4,-2;1,-2,1\right),$$

namely Immerkær estimator, which satisfies all those constraints mentioned above. Immerkær estimator has a simple form and high computational efficiency. In addition, it is a LSI system, and thus could be optimized in a closed-form mannner. Therefore, it is adopted in this paper for online noise estimation.

2.4. Image smoothing and details enhancement

In (2), increasing ${\alpha }_k$ is equivalent to increasing the value of the $k^{th}$ diagonal element of the matrix

$$[\mathbf{diag}\left(\mathbf{\alpha }\right)+\mathbf{C}\left({\mathbf{u}}^l\right)+\gamma \mathbf{G}\left(\sigma \left({\mathbf{u}}^l\right)\right].$$

(5)

Such a hyperparameter in the regularization term ${\alpha }_k{\left(a_k^l\right)}^2$ of (1) is set to punish $a_k^l$, the weight of $u_l^k$. This super parameter setting provides a choice of which frequency bands we want to enhance or suppress. In addition to tunning the weights channel-wisely, the values of $\left\{a_k^l\right\}$ could be increased/decreased uniformly to control the degree to which the input image is smoothed.

The meaningful area of an image always contains rich details and thus is dominated by the high frequencies energy. In EGF, these details are expected to be enhanced adaptively. It is assumed that $u_k^l$ is provided by performing filtering on $p_k^l$ with a high-pass bases from ${\mathbf{S}}^{n+1}$, and thus informative details of $p_k^l$ are mainly included in $u_k^l$. $u_k^l$ tends to be assigned a larger weight under the constraints, as the details (e.g. edges, textures, etc.) of $u_k^l$ are correlative with the content that in $p_k^l$, yielding a larger value of $\mathrm{Cov}(p_k^l,u_k^l)$ in (2).

$\mathbf{C}$ and $\mathbf{G}$ in (1) affect the output from the two aspects of the variance and noise level, respectively. With $\gamma >0$, the noise penalty term adaptively affect the adjustment of $a_k$. Assuming the noise level of $u_k^l$ is enlarged, the inner product of $\sigma \left(u_k^l\right)$ and itself increases, thereby raising the value of the k-th diagonal element of the matrix of (5), which is equivalent to enlarging the penalty of the noisy channel $u_k$.

In addition to adaptive balancing between detail enhancement and denoising, EGF can also facilitate balance denoising and edge preservation. Supposing $p^l$ is a flat area of the input, $\mathbf{Cov}(p^l,\mathbf{u})$ approaches zero, and as a results ${\mathbf{a}}_{1:n}^l$ become closer to zeros as well. According to the equation (2), $a_{n+1}^l$, the weight of the DC components, will be maximized. This means that the flat area will be strongly smoothed. If $p^l$ is an area with rich edges or textures, the covariances between $p^l$ and these maps in $\left\{u_k^l\right\}$ are of of higher values. Thus, the values of ${\mathbf{a}}_{1:n}^l$ solved are large, and the value of $a_{n+1}^l$ is close to zero. Therefore $p^l$ is reconstructed by those maps selected from $\left\{u_k^l\right\}$ instead of the DC component.

3. Experimental results and analysis

We first investigate the ability of EGF to enhance meaningful details during smoothing in both 1D and 2D cases, and then compare the performances of the EGF and GIF. In this paper, n is set to be 3, so ${\mathbf{S}}^{n+1}$ is a 4D space. As discussed in Section 2.2, $b_1$ is the impulse function, and therefore $u_1$ is just the input image. $b_4$ is the DC pass filter, so $u_4$ is a constant valued image. $b_2$ is set to be a high-pass filter, specifically using ${\nabla }^2{f}_{0.75}$, the $2_{nd}$ order difference of the Gaussian f with a variance of 0.75; $b_3$ as a low-pass Gaussian filter, $f_3$.

3.1. 1D case

Figure 3 shows the comparison of EGF and GIF on a 1D signal. The $1_{st}$ row is the input which has salt & pepper noise (green square) and Gaussian white noise (yellow square) added to the left and right, respectively. The $2_{nd}$ and $3_{rd}$ rows are the filtering results of EGF and GIF, respectively. GIF can achieve denoising while preserving dominant edges ($3_{rd}$ row). However, the details of these edges are suppressed to a certain extent. In contrast, the proposed EGF not only preserves these edges, but also enhances the details (red boxes). For Gaussian noise, both GIF and EGF achieve acceptable suppression, but the GIF has not suppressed them out thoroughly. In this case, if the regularization parameter of GIF is increased, the noise could be better suppressed, but at the same time, it will cause more damage to the edge. There is a similar result for salt & pepper noise. EGF is capable of suppressing both types of noise well, while keeping meaningful details to be enhanced. This advantage is achieved by using online noise level estimates and adaptive multi-spectrum bands integration.

3.2. 2D case

Figure 4 shows the filtering results of EGF on Lenna image. To investigate the ability of denoisng and details enhancement of EGF, frequency domain analysis is performed by comparing before and after spectral of three image patches. The enhancement spectrum is defined as $\left(\frac{log\mathcal{A}\left\{\mathcal{F}\left(\mathrm{g}\right)\right\}}{log\mathcal{A}\left\{\mathcal{F}\left(\mathrm{p}\right)\right\}}\right)·(log\mathcal{A}\left\{\mathcal{F}\left(\mathrm{g}\right)\right\}>log\mathcal{A}\left\{\mathcal{F}\left(\mathrm{p}\right)\right\}),$ where $\mathcal{A}\left\{\mathcal{F}(·)\right\}$ is the Fourier amplitude of the input. And suppression spectrum is defined in a similar way.

Patches in red and blue boxes have meaningful details that are expected to be enhanced during filtering for higher contrast. After EGF filtering, the details such as Lenna’s hair and eyes, as well as the brim of the hat, are sharpened and appear more clearly. From the analysis of the enhancement spectrum, One can find that the mid-range frequencies and some high-frequencies of these two image patches, corresponding to meaningful details, have been significantly enhanced.

Patch in green box is flat and has some degree of noise (see its log spectrum). After EGF filtering, the low-frequencies is enhanced (see its enhancement spectrum), while some mid-high frequencies are suppressed. In spatial domain, it indicates that the region has been further smoothed.

3.3. Comparison of EGF with GIF

Figure 5 shows the results of EGF and GIF on the input, with two zoomed patches of higher magnification placed under each result for comparison of details. To depict the filtering process of EGF, four weight maps, corresponding to $b_1$,...,$b_4$, are attached to $2_{nd}$ col., namely F, H, L, and D, respectively. Since we set ${\alpha }_1$ larger than that of other coefficients, EGF will suppress $a_1$ most, the coefficient of $u_1$, thereby forcing $u_2$,...,$u_4$ to play a greater role in reconstructing the input. $u_2$ contains more informative details, while $u_3$ and $u_4$ contain less noise. The edges of the image are mainly reconstructed by high-frequencies ($u_2$) and low-frequencies ($u_3$), whereas the flat area mainly by DC components ($u_4$). Since a high-pass filter is included as a basis in ${\mathbf{S}}^{n+1}$, EGF is capable to actively enhance the meaningful details, which is the main difference from traditional edge preserving filters. The risk that the enhancement of the high-frequencies may amplify the noise do exists in EGF, and the problem is alleviated through adding a regular term of online noise level evaluation.

GIF is a special case of the proposed EGF (n = 1) . The local combined filter of GIF is optimized within a 2D functional ${\mathbf{S}}^2$, where the two bases are all-pass and DC pass, respectively. GIF seeks for a tradeoff locally between the input and a constant valued image ($H\&L$ in Figure 6 are alwanys zero(blank)). Results illustrate GIF is capable to maintain the significant edges in smoothing, but fails at actively enhancing them and keep tiny details from being blurred.

Due to the introduction of regular terms for online noise level estimation, EGF is able to dynamically balance smoothing and detail enhancement according to noise conditions. In the first and the second rows of the Figure 6, different degrees of additive white Gaussian noise were added to the lower right part of each image. EGF(ours) can achieve adaptive noise suppression and detail enhancement. Observing their four weight maps, the noisy area of the image is mainly reconstructed by the energy of the low-frequency part ($L\&D$), and the information of the high-frequency part (H) is almost abandoned. The results suggest that EGF adaptively integrates the energy of different frequency bands according to the noise level. For GIF, however, the intensity of smoothing is determined by the variance of the current patch. If the area, corrupted by noise, has larger variance, it will not be sufficiently smoothed. In this case, if we increase the regularization coefficient $\alpha$ to force smooth heavily, more details of the image will be blurred along with noise (see last column of Figure 6). Tests were also conducted on heavy noisy images (see part B of Figure 6). It can be found that EGF keeps more details from being smoothed in filtering.

In imaging environments with low illumination, darker areas in the scene often exhibit stronger noise, which is due to the low signal-to-noise ratio in low illumination regions (as shown in the input image in Figure 7, where darker areas are contaminated by stronger noise). In other words, in low-light scenes, if there is significant variation in brightness across different areas, the spatial distribution of noise intensity in the resulting image also varies. Therefore, it is necessary to design appropriate algorithms to mitigate noise while minimizing disruption to texture regions. Such algorithms can be implemented in post-processing image enhancement software or in the Image Signal Processor (ISP) chip within the camera. However, denoising spatially non-uniform noisy images is a challenging task. For the GIF algorithm, the only adjustable parameter is ${\alpha }_1$. As shown in the figure, we attempted to gradually adjust ${\alpha }_1$ from ${0.05}^2$ to ${0.2}^2$. However, we found that the GIF algorithm cannot achieve a good balance between denoising and preserving weak textures. In other words, when a stronger noise suppression is desired, it inevitably results in severe disruption (filtering out) of weak texture areas in the image. This is primarily due to the GIF algorithm’s reliance solely on variance to evaluate local image information. Since noise areas and texture areas may have similar variances, noise needs to be filtered out while weak textures need to be preserved. In contrast, our proposed EGF algorithm introduces local noise estimation and benefits from the filtering framework presented in this paper, thereby demonstrating a certain capability to differentiate between noise and texture areas and to adaptively process these areas separately.

Next, we further conducted quantitative analysis. For the aforementioned images, we considered extracting noise patches and text patches from the images and conducted a comparative quantitative analysis. We computed the Average Gradient Response (AGR) for these two patches before and after filtering by considering the average gradient response level. We used the Laplacian operator to convolve the aforementioned two areas and calculated the mean value of the convolution response amplitude to represent the average gradient response level. As shown in Table 1, the AGR of the noise area and the text area in the original input image are $59.90\%$ and $10.45\%$, respectively, indicating a high noise level in the noise area. Filtering using the GIF algorithm shows that as ${\alpha }_1$ increases, the noise level in the noise area can be reduced from $22.17\%$ to $80.97\%$. However, as ${\alpha }_1$ increases, the text area is also significantly suppressed, with suppression rates exceeding $45\%$ and reaching as high as $88.71\%$. Moreover, regardless of the choice of ${\alpha }_1$ we cannot achieve good noise suppression while preserving texture areas from severe disruption. In contrast, our proposed EGF algorithm can achieve a noise suppression of $83.02\%$ while causing only $29.57\%$ suppression in the texture area.

4. Conclusion

In this paper, we present a new paradigm of image filter design. In this paradigm, filters that corresponding to different spectrum bands are employed as the bases to form a functional space where the optimal locally based filter is solved based on optimization. By introducing noise estimation as regular terms, the filter achieves joint functions of both smoothing and detail enhancement. In addition to Gaussian filter used in this paper, other forms of filter could also be adopted as the bases. There are two problems worthy of further investigation. One is to consider the selection of the bases of the functional subspace as an optimization problem as well as solving the optimal bases; the other is to find a more favorable prior knowledge about a "good" map as a regular term.