Review of Matrix Rank Constraint Model for Impulse Interference Image Inpainting

1. Introduction

In machine vision applications, images often suffer from impulse interference due to various factors, such as pulse noise caused by detector pixel failure in the camera or loss of storage elements in imaging [1]. Satellite images, unmanned aerial vehicle (UAV) images, etc. generally have local smoothness, so their two-dimensional representation matrix usually has obvious low rankness. Low-rank prior information has an excellent performance in image denoising [1], inpainting [2,3], reconstruction [4], deblurring [5], and other signal optimization processing fields. In the existing low-matrix-rank based image inpainting methods, the low-rank prior information is mainly divided into: the low rank of the signal itself, such as the inherent low rank of the matrix, the similarity of local map block information [6], the similarity between frames of video [7], etc. The Hankel-like structure low-rank matrix can be constructed in the Fourier domain by using the annihilating filter relationship [2,4,8]; The high-order tensor rank can be obtained under various tensor decomposition frameworks, such as CANECOMP/PARAFAC (CP), Tucker [9,10], tensor train (TT) [11], and tensor singular value decomposition (T-SVD) [12,13].

Besides low-rank prior, early image denoising methods assumed that the image had sparse representations in its transformation domains, such as difference domain, wavelet domain, etc. [14,15,16,17], and then constrained the sparse prior information to recover the image from the noise. Due to the effectiveness of low rank and sparsity in constrained image optimization problems, image processing schemes that combine constrained sparsity with low-rank prior information have been continuously proposed [18,19,20]. Some image denoising methods use decomposition models of low-rank and sparse components, such as the robust principal component analysis (RPCA) method [21,22], aimed at separating low-rank images from sparse interference images. With the research and development of tensor decomposition tools in the field of mathematics, such as t-SVD, TT decomposition etc. [11,23,24], related image or video optimization applications based on low-rank tensor are also being developed [25,26,27,28,29].

In addition to using low-rank prior information to construct the constraint model for impulse interference image inpainting, many theories, methods, and technologies in the field of signal processing can be used to solve image inpainting problems, such as various matrix/tensor completion theories in mathematics [30,31,32], finite innovation rate (FIR) theory [33], image and video enhancing (such as Hankel like matrix based technology [22]), denoising schemes (such as famous BM3D image denoising technology [34], non-local TV denoising technology [14,15,35]), etc. Various tensor-decomposition-based completion methods, convex optimization schemes, and fast optimization algorithm research systems can also be used to optimize image inpainting methods.

This paper uses the low-rank property of the image matrix to optimize the image inpainting modeling and algorithm under three kinds of pulse interference. Image inpainting modeling schemes based on nuclear norm, truncated nuclear norm, weighted nuclear norm, and matrix-factorization-based F norm are reviewed, and corresponding optimization iterative algorithms, such as TSVT_ADMM algorithm, WSVT_ ADMM algorithm, and UV_ ADMM algorithm are given. The experimental results of various inpainting methods are displayed visually and numerically, and a comparative analysis is given.

The structure and content of this paper are arranged as follows: Section 1, introduction; Section 2, based on the matrix low-rank constraint inpainting model and solution algorithm; Section 3, experimental comparison; The last section is conclusions.

2. The Matrix Low-Rank Constrained Inpainting Model and Its Solution Algorithm

Image inpainting models based on low-rank matrix are generally as follows:

(1)

where X represents the image to be recovered (X ∈ R² grayscale value image, X ∈ R³ RGB image, grayscale value video, etc.). Θ_Ω represents the interference operator, in which the set of interference pixel positions is Ω. $\widehat{X}$ represents the optimal solution. Y represents the interfered image. ε represents the error, generally set as a small constant value matrix, such as constant 10^-14. Φ represents low-rank transformation operation. ΦX is to transform X into a matrix or tensor with low ranks, such as a low-rank matrix transformed by the similar blocks of local images, the low-rank structure matrix [1,2,4,22] transformed by the relationship of the annihilating Filter, and the low-rank matrix [7] transformed by the similarity between frames. If the videos or RGB images are treated as third-order or higher-order tensors, the rank property may come from the tensor Tucker rank [36], TT rank [26], etc. Under the interference of impulse noise, Θ_Ω operator generally has three representations.

The first is random valued impulse noise (RVIN)[1].

The value in V is random, and its value range is within the range of X's pixel value, such as the range of 0~255, or the range of normalized 0~1. p is the interference rate, that is, the percentage of the number of interfering pixels in the total number of pixels in the image.

The second is salt and pepper noise, a special case of RVIN [1],

where V_max is the maximum value of salt pepper noise and V_min is the minimum value of salt pepper noise.

In addition, random pixel loss is also a typical problem in the research field of image repair [2,6,9], namely

The low-rank property is another form of sparsity essentially. Sparse constraints on matrices essentially minimize the zero norm of matrix elements, while low-rank constraints minimize the zero norm of singular values of the constraint matrix. The low-rank constraint on a matrix is actually the l₀ norm constraint on the singular values of the matrix, i.e. $\underset{X}{\min }rank\left(\mathsf{\Phi }X\right)⟺\underset{X}{\min }{‖\mathsf{\Phi }X‖}_0$. Since $\underset{X}{\min }{‖\mathsf{\Phi }X‖}_0$ is nonconvex, the l_p norm form $\underset{X}{\min }{‖\mathsf{\Phi }X‖}_{\mathrm{p}}$ is commonly used for convex substitution[37], where 0 ≤ p ≤ 1, ${‖\mathsf{\Phi }X‖}_{\mathrm{p}}={\displaystyle \sum _{i=1}^n}{{\sigma }_i}^p$, and σ_i are singular values of matrix ΦX of size n₁ × n₂, n = min(n₁, n₂). The special case of the l_p norm is the nuclear norm ${‖\mathsf{\Phi }X‖}_{\mathrm{*}}={\displaystyle \sum _{i=1}^n}{\sigma }_i$, which means p = 1. Whether the low-rank constraint form used can accurately convex approximation has a significant impact on the repair effect. Let ${‖\mathsf{\Phi }X‖}_p={\displaystyle \sum _i^n}{g}_p\left({\sigma }_i\right)$, where function g_p(σ_i) = ${{\sigma }_i}^p$, 0 ≤ p ≤ 1. For the l₀ and l_p norm, the function g_p(σ_i) is

Normalize σ_i within the range of 0-1, and plot the curves of function g_p(σ_i) at p=0, 0.3, 0.5, 0.7, and 1. The visualization of convex approximation is shown in Figure 1. It can be seen that the smaller the p is, the closer the convex approximation function g_p(σ_i) curve is to the l₀ norm curve.

As the simplest convex substitution of l₀ norm, the nuclear norm is the most common in low rank constraint modeling. To further improve the accuracy of low-rank approximation, we can use the weighted l₁ norm of the singular values of the matrix, that is, the weighted nuclear norm [38,39,40,41], or use the truncated nuclear norm [42,43,44,45] to replace the nuclear norm. Common regularization constraint schemes for low-rank matrices are summarized as follows.

2.1. Nuclear norm ‖X‖_∗

For example, we use the minimized nuclear norm as a low rank constraint to establish an image inpainting model, as follows.

(2)

Y is the impulse interference image of size n₁ × n₂. The regularization parameter λ is introduced to convert model (2) into an unconstrained form:

(3)

Three algorithms can solve equation (3). The commonly used algorithm is the singular value shrinkage/threshold (SVT) algorithm [46].

The SVT algorithm is shown below.

First, perform singular value decomposition for Y, U∑V^H = SVD(Y), ∑ = diag( {σ_i}_1≤i≤n ), where diag( . )is the diagonal matrix operation of the elements, n = min(n₁, n₂). Then, a soft threshold operation D_λ(σ_i) = max(0, σ_i − λ) is performed on the singular values [47]. Then, set ∑_SVT = diag( {D_λ(σ_i)}_1≤i≤n ). Finally, we obtained the solution results $\widehat{X}$ = U∑_SVTV^H.

Jain P. proposed the SVP algorithm for solving model (2) problem [48]. With the development of large-scale data processing and distributed computing, the alternating direction method of multipliers (ADMM) algorithm has become the mainstream optimization algorithm [49]. When using the ADMM algorithm to solve (3), it needs to first introduce auxiliary variables Z = ΦX and residuals L to transform model (3) into multiple sub-problems for an iterative solution:

where, ρ > 0 is the introduced penalty parameter, and the SVT method is used to solve the sub problem $\hat{Z}$.

In this paper, we use the SVT algorithm, SVP algorithm, and ADMM algorithm [50,51] to solve the nuclear-norm-based image inpainting model, which corresponds to name as the SVT method, SVP method, and n_ADMM method respectively. The details of the SVT algorithm, SVP algorithm, and n_ADMM algorithm are shown in Table 1, Table 2 and Table 3 respectively.

2.2. Weighted nuclear norm ${\displaystyle \sum _{i}fun({\sigma }_i)}$

The weighted nuclear norm ${\displaystyle \sum _{i}fun({\sigma }_i)}$ is a scheme that uses weighted singular value constraints to approximate l₀ constrained singular values [38,39,40,41]. It is a balanced constraint scheme that makes large singular values smaller and small singular values larger. It can be more accurate than the nuclear norm (i.e., l₁ constraints on singular values), where fun( . ) is the weighted function of each singular value σ_i of matrix ΦX, and [U, ${diag\left\{{\sigma }_i\right\}}_{{i=1:\min (n}_1,n_2)}$, V] = SVD(ΦX). We use the weighted nuclear norm as a low-rank constraint to establish an image inpainting model, as follows.

(4)

Then, we introduce the regularization parameter λ and converted model (4) into an unconstrained form:

(5)

There are many kinds of weighting functions fun( . ), and the p-norm (0<p<1) is the simplest weighting scheme, namely g_p(σ_i) = fun(σ_i). Reference [39] reviewed various weighting functions that approximate the l₀ norm of singular values, such as SCAD [52], MCP [53], Logarithm [54], Geman [55], Laplace [56,57], etc., of which the Logarithm scheme is the most classic. In the experiment part of this paper, we choose the Logarithm scheme for comparison. The weighting function in the Logarithm scheme is shown below.

(6)

where γ > 0 is a parameter that is determined based on experience.

The simplest and most direct solution for model (4) is the weighted SVT (WSVT) algorithm. Set the weight w_i = fun(σ_i), i = 1, 2, …, n, and then $\widehat{X}$ = U∑_WSVTV^H, where ∑_WSVT = diag({D_λ(w_iσ_i)}_1≤i≤n).

We use ADMM to solve the weighted nuclear norm image inpainting problem (5). We introduce auxiliary variables Z = ΦX and residuals L to transform model (5) into multiple sub-problems for iterative solution:

(7)

where ρ > 0 is the introduced penalty parameter, and the WSVT algorithm is used to solve the sub problem $\hat{Z}$. The combination of the weighted SVT algorithm and ADMM algorithm can obtain more accurate iterative estimation. We use the ADMM algorithm to solve the weighted-nuclear-norm-based image inpainting model (7), and name it WSVT_ADMM method. The details of WSVT_ADMM algorithm used to solve model (7) are shown in Table 4.

2.3. Truncated nuclear norm

In general, the singular value curve of a low-rank matrix exhibits an exponential extreme decay trend from large to small, and the singular value values sorted backward will approach 0. Therefore, the minimization of the nuclear norm is mainly to constrain the minimization of large singular values. To fully utilize the small singular values, a truncated nuclear norm minimization scheme can be used. The purpose of truncated nuclear norm minimization is to constrain the minimization of small singular values [42,43,44,45]. We use the truncated nuclear norm as a low-rank constraint to establish an image inpainting model, as follows.

(8)

where ${\mathcal{T}}_r$ is a truncation operation that extracts the first r larger diagonal elements in the diagonal matrix U^HΦXV, and ‖ΦX‖_∗ − ${\mathcal{T}}_r$(U^HΦXV) means that the first r larger diagonal elements in the diagonal matrix U^HΦXV are zeroed, and the last r smaller diagonal elements are retained [58,59]. We introduce a regularization parameter λ and converted (8) into an unconstrained form:

(9)

U, V is the truncated left and right singular value vector group matrix of ΦX. The essence of truncated nuclear norm minimization is to minimize the sum of smaller singular value elements of the constrained low-rank matrices.

The truncated-nuclear-norm-based model can be solved by APGL or ADMM algorithm. This paper combines the ADMM algorithm with the SVT algorithm to solve the truncated-nuclear-norm-based image inpainting model (9), and abbreviates it as the TSVT (Truncated SVT) algorithm. The details of the TSVT algorithm used to solve model (9) are shown in Table 5.

2.4. The F norm of UV matrix factorization

The process of solving the nuclear norm minimization problem involves time-consuming matrix singular value decomposition. Early Srebro N. [60] proposed the property ${‖X‖}_{*}=\underset{{UV}^H=X}{\min }\frac{1}{2}({‖U‖}_F^2+{‖V‖}_F^2)$ and proved it. Later, in many applications, the F norm of UV matrix factorization was used instead of the nuclear norm to simplify the calculation time [1,61,62,63,64]. We use the minimized F norm of UV matrix factorization as a low-rank constraint to establish an image inpainting model, as follows.

(10)

Then, we introduce the regularization parameter λ and penalty parameter ρ > 0 to convert model (10) into an unconstrained form:

(11)

where L is the residual variable. The initial values of U and V can be initialized by the LMaFit method [2,65]. The model (11) is commonly solved by the ADMM algorithm, and we name it UV_ ADMM method. The details of UV_ ADMM algorithm are shown in Table 6.

Compared with the n_ADMM method, the F-norm of UV-matrix based UV_ ADMM method avoids the time-consuming SVD in each iteration, making it more suitable for large matrix modeling with low-rank constraints. This method and the weighted nuclear norm method are commonly used in low-rank matrix constrained models.

The above model and its solving algorithm are summarized in Table 7. In addition, other algorithms that can solve the above model. For example, commonly used algorithms in sparsity-solving models. Sparsity constraint on signal refers to minimization of l₀ norm of signal elements, while low-rank constraint on signal refers to minimization of zero norms of the singular value of the signal matrix. Therefore, the optimization solution based on low-rank constraint model has a lot in common with the optimization solution based on the sparse constraint model in solving the algorithm. Iterative optimization algorithms based on sparse constraint models can be applied to optimization solutions based on matrix low-rank constraint models, such as convex relaxation algorithms, which find a sparse or low-rank approximation of signals by transforming nonconvex problems into convex problems through iterative solutions. Among them, the CG algorithm, IST algorithm [66], Split Bregman algorithm [67], and MM (major minimize) algorithm [58,68] can be flexibly changed according to different optimization models.

3. Comparative Experiments

In this section, we conduct a comparison of the above methods for solving satellite image inpainting problems. We simulated the impulse interference on satellite images with an interference rate of 30%1. The three kinds of impulse interference were: A. Random impulse interference; B. Spicy salt impulse interference; C. Random pixels missing. The satellite images in this paper are sourced from https://captain-whu.github.io/ DOTA/dataset.html public dataset DOTA v.2.0, images provided by China Resources Satellite Data and Application Center, satellite GF-1, satellite GF-2, etc. The comparison methods: SVT, SVP, n_ADMM, TSVT_ ADMM, WSVT_ ADMM, and UV_ ADMM. For a fair comparison, every method is conducted with its optimal parameters to ensure every method has the best performance.

The relative least normalized error (RLNE) and the structural similarity (SSIM) [69] are used as image inpainting quality indicators. The RLNE is an index based on the error between pixels, and the SSIM index is more consistent with human visual perception in image visual evaluation. Generally, the larger the SSIM value is, the better the image inpainting quality is. All simulations were carried out on Windows 10 and MATLAB R2019a running on a PC with an Intel Core i7 CPU 2.8GHz and 16GB of memory.

A gray satellite image and its singular values curve are shown in Figure 2a,b respectively. The singular values of the image are descending rapidly from large to small, and most of them tend to be zero. This indicates that the image has the characteristics of low rank. The three examples of impulse interference satellite images are shown in Figure 3, where the original image is the image in Figure 1a. It can be seen that the 30% interference rate has caused obvious information loss on the building shape, layout, gray value shading, and other features in the original image.

The comparison of the average values (RLNE, SSIM, and running time) of six image inpainting methods under the interference of random impulse, salt and pepper noise, and pixel missing is shown in Table 8. The visual comparison of the six image inpainting methods under the interference of salt and pepper noise is shown in Figure 4.

Based on the above visual and numerical comparison, we analyze the experimental methods below.

The matrix rank constraint method based on the F norm of UV-matrix-factorization (i.e. UV_ADMM method) is equivalent to the method based on nuclear norm constraint (i.e. n_ADMM method) in terms of effectiveness. Overall, the n_ADMM method is slightly better, increasing the RLNE index by about 0.3% and the SSIM index by 0.3~1.

Due to that the nuclear-norm-based SVT, SVP, n_ADMM methods, the weighted nuclear norm-based WSVT_ADMM methods, and the truncated nuclear norm-based TSVT methods all involve time-consuming SVD calculations in each iteration, the UV_ADMM method based on UV matrix factorization F norm have absolute advantages in terms of runtime. However, the UV_ADMM method did not achieve better accurate results compared to other methods, because the UV_ADMM method needs to initially set the estimated rank. Such as using the LMaFit method to initialize the rank. However, the rank is initialization estimated and the accuracy of the rank is not high, which leads to inaccurate low-rank constraints. So, this UV-matrix-factorization-based method is more commonly used for large-scale low-rank matrix calculations, and it can greatly reduce the inpainting time of large-scale matrix optimization due to avoiding SVD of each iteration.

Since the weighted and truncated nuclear norm can better convex approach the singular value l₀ norm, the WSVT_ADMM and TSVT methods are significantly better than that of the nuclear-norm-based methods (SVT, SVP, n_ADMM) in terms of the accuracy of inpainting.

4. Conclusions

In the application of machine vision, satellite images may suffer from three forms of impulse noise interference. In this paper, we use the low-rank characteristics of the image matrix to optimize and repair the images under the three kinds of impulse interference and provide the optimization algorithm. Firstly, image inpainting modeling schemes based on nuclear norm, truncated nuclear norm, weighted nuclear norm, and matrix factorization F norm are reviewed. Then, the corresponding optimization iterative algorithm is given, such as the TSVT_ ADMM algorithm, WSVT_ ADMM algorithm, UV_ ADMM algorithm, etc. Finally, the experimental results of various matrix-rank-constraint-based methods are presented visually and numerically, and a comparative analysis is given. The experimental results show that all the mentioned matrix-rank-constraint-based methods can repair images to a certain extent and suppress certain interference noise. Among them, methods based on weighted nuclear norm and methods based on truncated nuclear norm can achieve better repair effect, while methods based on matrix factorization F norm take the shortest time and can be used for large-scale matrix low-rank calculation.