Low Tensor Rank Constrained Image Inpainting Using a Novel Arrangement Scheme

1. Introduction

Image inpainting refers to the process of completing missing entries or restoring damaged regions of an image. It is a typical ill-posed inverse problem, generally solved by exploiting the image priors [1,2], such as smoothness, sparsity, and low rankness. In recent years, tensor analysis including tensor low-rank decomposition and tensor completion, has attracted increasing attention [3,4,5,6]. A color image itself is an order-3 tensor, or it can be used to construct a high order (greater than 3) tensor, then the image inpainting problem becomes a tensor completion problem. A tensor is more challenging to analyze than a matrix due to the complicated nature of higher-order arrays [7]. We can constrain the low tensor rank to recover the missing pixels. The effectiveness relies on the tensor rank. The lower the tensor rank is, the better the recovery results are. Thus, finding ways to decrease the tensor rank is essential in the tensor completion problem. Unlike matrix rank, the definition of tensor rank is not unique, and relates to the tensor decomposition scheme.

Low tensor-rank completion methods can be categorized according to the tensor decomposition frameworks they use [8]. The traditional tensor decomposition tools include CANDECOMP/PARAFAC (CP), and Tucker decomposition [8,9]. The recently proposed decomposition frameworks include tensor singular value decomposition (t-SVD) [10,11,12], tensor train (TT) decomposition [13,14], tensor tree (TTR) decomposition [6,15] etc. As we know, CP rank is hard to estimate. Tucker rank is multi-rank, whose elements are the ranks of mode-n matrices which are highly unbalanced. TT rank is also multi-rank, whose elements are the ranks of TT matrices. For a high-order tensor, the most TT matrices are more balanced than the mode-n matrices. Since the matrix rank minimization is only efficient when the matrix is balanced, TT decomposition is more suitable for describing global information of high-order tensors than Tucker decomposition. T-SVD defines the tubal rank of the high order tensor, which can be easily estimated according to a fast Fourier-based method. The tubal rank has been shown more efficient than the matrix-rank and Tucker multi-rank in video applications [16,17,18]. TTR rank is essentially equivalent to Tucker multi-rank.

Many popular tensor-completion methods have applied the traditional CP or Tucker decomposition on color image inpainting. Some recent works exploited the sparse Tucker core tensor and nonnegative Tucker factor matrices for image restoration [7,19,20]. Some works constrained the low rankness of the mode-n matrix caused by decomposition of a color image for inpainting [21,22]. Since low-tensor-rank constraint cannot fully capture the local smooth and global sparsity priors of tensors, some works combine Tucker and total variation (TV). The SPCTV (smooth PARAFAC tensor completion & total variation) method [23] used the PD (PARAFAC decomposition, a derivation of Tucker decomposition) framework, and constrained the TV (total variation) on every factor matrix of PD respectively. Some works combined the constraints of the low rankness of every mode-n matrix and the TV regularization on every mode-n matrix for color image inpainting [24,25]. Some works proposed data restoration methods based on Bayesian tensor completion [26,27,28,29].

The afore-mentioned methods all take the color image as an order-3 tensor directly and haven’t deeply explored the potential low-rank prior to a color image. Since TT decomposition is efficient for higher-order tensors, the TMac-TTKA method [30] first used the Ket augmentation (KA) scheme to permute the image to a high order data, then proposed the optimal models by enforcing low TT rankness. The KA scheme is proven to be efficient for improving the accuracy of color image/video inpainting and dynamic MR image reconstruction in TT rank based completion methods [30,31,32,33]. As far as we know, the KA scheme is the only one used to permute data into a high order data.

This paper aims to deeply explore the potential low-rank structure of the image and to find an efficient way to apply the SVD, t-SVD, and TT decomposition in the image inpainting problems. The contributions of our work are summarized as follows:

• First, we developed a novel rearrangement named as quarter augmentation (QA) scheme for permuting the image into three flexible forms of data. The first flexible QA scheme can permute an image into an unfolding matrix (with a low matrix rank structure). The second and the third flexible QA schemes can permute the color image into a balanced 3-order form of data (with low tubal rank structure) and a higher-order form of data (with low TT rank structure) respectively. Since those developed schemes are designed to exploit the internal structure similarity of the original data as much as possible, the rearranged data has the corresponding kind of low-rank structure.
• Second, based on the above QA scheme, we developed three image inpainting models that exploit the unfolding matrix rank, tensor tubal rank, and TT multi-rank of the rearranged data respectively for solving the image inpainting problem.
• Lastly, three efficient ADMM algorithms were developed for solving the above three models. Compared with numerous close image inpainting methods, the experimental results demonstrated the superior performance of our methods.

The remainder of this paper is organized as follows. In section II, we give the related work. In section III, we mainly introduce the proposed methods. Section IV the experimental results and analyses. The conclusion is given in section V.

2. Related work

In this section, we briefly introduce the KA scheme, the t-SVD decomposition, and tensor train decomposition. Notations and definitions are summarized in Table 1.

2.1. Ket Augmentation

The Ket Augmentation (KA) scheme was originally introduced by Latorre in [34] for casting a grayscale image into the real ket state of a Hilbert space. Bengua etc. [30] used KA to reshape a low-order tensor e.g. a color image to a higher-order tensor and proved that KA is efficient in improving the accuracy of the recovered image in TT-based completion.

Figure 1 shows the operation of KA for an 8 × 8 matrix [31,32,33,34]. By the KA scheme, the 8 × 8 matrix can be turned into a 3-order tensor of size 4 × 4 ×4. As well, the KA scheme can turn a 3-order tensor size of $x^N\times y^N\times N_3$ into an $\left(N+1\right)$-order tensor with a size of $xy\times xy\times :::\times xy\times N_3$.

2.2. T-SVD Decomposition

Definition 1

t-product [35]. For $\mathcal{A}\in R^{n_1\times n_2\times n_3}$ and $\mathcal{B}\in R^{n_2\times n_4\times n_3}$, the t-product $\mathcal{A}\ast \mathcal{B}=\mathcal{C}$ is a tensor of size $n_1\times n_4\times n_3$ where $\mathcal{C}\hspace{0.17em}(i,j,:)$ is given by ${\displaystyle {\sum }_{k=1}^{n_2}\mathcal{A}\hspace{0.17em}(i,k,:)}\circ \mathcal{B}\hspace{0.17em}(k,j,:)$. $\circ$denotes the circular convolution between the two vectors, and $i=1,\hspace{0.33em}2,\cdots ,\hspace{0.33em}{n}_1$, $j=1,\hspace{0.33em}2,\cdots ,\hspace{0.33em}{n}_4$.

The t-SVD of $\mathcal{A}\in R^{n_1\times n_2\times n_3}$ is given by

$$\mathcal{A}=\mathcal{U}\ast \mathcal{S}\ast {\mathcal{V}}^T$$

where $\mathcal{U}$ and $\mathcal{V}$ are orthogonal tensors of size $n_1\times n_1\times n_3$ and $n_2\times n_2\times n_3$ respectively. $\mathcal{S}$ is a rectangular f-diagonal tensor of size $n_1\times n_2\times n_3$ and * denote t-product [35], $T$ denotes tensor transpose defined in [35].

Figure 2 depicts the t-SVD of an order-3 tensor [4,36]. Tensor rank defined in t-SVD is tensor tubal rank, which is the number of nonzero singular tubes in $\mathcal{S}$. [36] proposed the fast Fourier-based method to calculate the tubal rank, and used tensor nuclear norm (TNN) as the convex relaxation of the tensor tubal rank.

$${‖\mathcal{A}‖}_{TNN}={‖blockdiag(\overline{\mathcal{A}})‖}_{*}$$

where $\overline{\mathcal{A}}\hspace{0.17em}=\hspace{0.17em}fft\hspace{0.17em}(\mathcal{A},\hspace{0.33em}[\hspace{0.17em}]\hspace{0.17em},\hspace{0.33em}3)$ is the tensor obtained by applying the 1D FFT along the third dimension of $\mathcal{A}$, ${‖‖}_{*}$ denotes nuclear norm, and

$$blockdiag(\overline{\mathcal{A}})=\left[\begin{array}{cccc}{\overline{\mathcal{A}}}^{(1)}& & & \\ & {\overline{\mathcal{A}}}^{(2)}& & \\ & & \ddots & \\ & & & {\overline{\mathcal{A}}}^{(I_3)}\end{array}\right]$$

2.3. Tensor Train Decomposition

$\mathcal{A}(i_1,i_2,\cdots i_n,\cdots i_N)={\mathcal{U}}_1(:,i_1,:){\mathcal{U}}_2(:,i_2,:)\cdots {\mathcal{U}}_n(:,i_n,:)\cdots {\mathcal{U}}_N(:,i_N,:)$

Given a tensor $\mathcal{A}\in R^{I_1\times I_2\times \cdots I_N}$, tensor train (TT) decomposition [13,14] can decompose it to $N$ order-3 tensors ${\mathcal{U}}_n\in R^{S_n\times I_n\times \cdots S_{n+1}}$, $n=1,\cdots ,N$. The tensor rank defined in TT decomposition is a multi-rank i.e. $(S_1,S_2,\cdots ,S_{N+1})$, which is combined with the second-dimensional size of each ${\mathcal{U}}_n$. The details of TT decomposition are shown in the following formula and Figure 3 [31,32,33].

The widely used way to find TT rank is to estimate the rank of each TT matrix [37] as the element of $(S_1,S_2,\cdots ,S_{N+1})$. The TT matrix ${\mathcal{A}}_{[\mathrm{n}]}$($n=1,\cdots ,N-1$) with rank $S_n$ is the mode-$\left(1,2,\cdots ,n\right)$ matricization of the tensor with the size of $m\times h$, where $m={\displaystyle \prod _{l=1}^n{I}_l}$, $h={\displaystyle \prod _{l=n+1}^N{I}_l}$.

3. Methods

3.1. Quarter Augmentation

To deeply explore the more efficient low-rank structure of an image, we develop a novel rearrangement scheme named as quarter augmentation (QA) scheme to turn a color image into other forms of data. The QA schemes can maintain the internal similarity of the original image in the rearranged data.

The basic QA scheme: For example, as shown in Figure 4 (a), M is a 2D matrix ($8\times 8$). We first extract the entries of M every other row and column to get four smaller matrices. Each smaller matrix with a size of $4\times 4$. Then we place these four smaller matrices along the third dimension in a designed order. Lastly, a 3D tensor of size $4\times 4\times 4$ is obtained from the $8\times 8$ matrix M without changing the total number of entries. The entries in the four smaller matrices are labeled as the MATLAB notation $(:,:,1)$, $(:,:,2)$, $(:,:,3)$ and $(:,:,4)$ respectively. If M is smooth (most images satisfy), the four smaller matrices are similar in structure due to the adjacent entries.

Applying the basic QA scheme on the single Lena image, the Lena image can be divided into 4 smaller Lena images, and as shown in Figure 4 (b) the four smaller Lena images are similar to each other. In Figure 4 (c), the pixel values curves of the four smaller images have overlapped into one curve. We can say that the similarity of local image structure is mainly maintained by the basic QA scheme.

Under this basic QA scheme, three flexible QA schemes are proposed for permuting the image into three flexible forms of data. The three flexible QA schemes can enhance the low-rankness for an image by matrix SVD, tensor train decomposition, and tensor-SVD respectively. Then, by exploiting the flexible QA schemes, three low-rank constrained methods which use the TT rank, tubal rank, and matrix rank as constrained priors respectively are exploited for image inpainting.

The three flexible QA schemes and methods are described in detail in the following three sections.

3.2. Method 1: The Low Unfolding Matrix Rank-Based Method

The unfolding method is widely used to permute the order-3 video or dynamic magnetic resonance images into an unfolding matrix, and then exploit the low rankness of this matrix for data reconstruction [21,38]. The unfolding matrix has a low-rank structure because of the similarity of every slightly changed slice along the time dimension.

We try to dig out the potential low unfolding-matrix rankness of a color image by a flexible QA scheme, and we call this scheme the first flexible QA scheme.

Take a 256×256×3 Lena image as an example, as shown in Figure 5 (a), we first permute the image into the 3-order tensor size of 32×32×192 by the basic QA scheme, then unfold the similar slices of this 3-order. Lastly, the balanced1 unfolding matrix size of 1024×192 is obtained. Since the slices (32×32) in the 3-order tensor are similar, the unfolding matrix is low rank, as shown in Figure 5 (b). In practice, the size of the designed unfolding matrix should be balanced such that the minimization of the unfolding matrix rank is efficient.

We exploit the low unfolding matrix rank in image inpainting and give the low unfolding matrix-rank-based model as follows.

$$\underset{\mathrm{X}}{\min }\hspace{1em}{‖\mathrm{{\rm M}}({\mathsf{\Phi }}_{1}X)‖}_{*}\hspace{1em}\mathrm{subject} \mathrm{to}\hspace{1em}X(\mathrm{i},\hspace{0.33em}\mathrm{j})=Y(\mathrm{i},\hspace{0.33em}\mathrm{j}),\hspace{0.33em}\forall (\mathrm{i},\hspace{0.33em}\mathrm{j})\in \mathsf{\Omega }$$

(1)

where $X$ denotes the image to be recovered, ${\mathsf{\Phi }}_1$ denotes the operator of permuting the image into a suitable 3-order tensor by multiple basic QA schemes. $\mathsf{{\rm M}}$ denotes the operator of the unfolding process, which unfolding every slice along the third dimension of the 3-order tensor ${\mathsf{\Phi }}_{1}X$. $\mathsf{\Omega }$ is the position without painting, $Y$ is the painted image with damaged entries at the positions ${\mathsf{\Omega }}^{ℂ}$.

To reduce the computational complexity, in the model (1), the following SVD-free approach [39,40] is exploited to constrain the low rankness of the unfolding matrix $\mathsf{{\rm M}}({\mathsf{\Phi }}_{1}X)$ instead of the nuclear norm.

$$\underset{U_{}{V}_{}^H=\mathsf{{\rm M}}({\mathsf{\Phi }}_{1}X)}{\min }\frac{1}{2}\left({‖U_{}‖}_F^2+{‖V‖}_F^2\right)={‖\mathsf{{\rm M}}({\mathsf{\Phi }}_{1}X)‖}_{*}$$

(2)

Besides, since total variation (TV) has been proved as an effective constraint of smooth prior [41,42], incorporate model (1) with 2D TV to exploit the local smooth priors of visual image data. Then, the image inpainting model (1) turns to the following.

$$\begin{array}{l}\underset{U,\hspace{0.33em}V,\hspace{0.33em}X}{\min }\frac{1}{2}\left({‖U‖}_F^2+{‖V‖}_F^2\right)+\frac{\beta }{2}{‖X‖}_{TV}\hspace{1em}\\ \mathrm{subject} \mathrm{to}\hspace{0.33em}\hspace{0.33em}X(\mathrm{i},\hspace{0.33em}\mathrm{j})=Y(\mathrm{i},\hspace{0.33em}\mathrm{j}),\hspace{0.33em}\forall (\mathrm{i},\hspace{0.33em}\mathrm{j})\in \mathsf{\Omega },\hspace{1em}\mathsf{{\rm M}}{\mathsf{\Phi }}_{1}X=U{V}^H\end{array}$$

(3)

where $\beta$ is the regularization parameter.

We conduct the algorithm by alternating direction method of multipliers (ADMM) for solving the low unfolding matrix rank and TV-based image inpainting model (3). Firstly, introduce an auxiliary variable $Z=DX$, where $D$ is the finite difference operator, and then rewrite (3) as the unconstrained convex optimization problem (4).

$$\begin{array}{l}\underset{U,\hspace{0.33em}V,\hspace{0.33em}\mathsf{\Lambda },\hspace{0.33em}L,\hspace{0.33em}Z,\hspace{0.33em}X}{\min }{\tau }_{\mathsf{\Omega }}(X)+\frac{1}{2}\left({‖U‖}_F^2+{‖V‖}_F^2\right)+\frac{{\rho }_1}{2}{‖\mathsf{{\rm M}}{\mathsf{\Phi }}_{1}X-U{V}^H+\mathsf{\Lambda }‖}_F^2\\ \hspace{1em}\hspace{0.33em}\hspace{0.33em}+\frac{\beta }{2}{‖Z‖}_1+\frac{\beta {\rho }_2}{2}{‖DX-Z+L‖}_F^2\end{array}$$

(4)

where ${\tau }_{\mathsf{\Omega }}(X)$ denotes the indicator function:

 ${\tau }_{\mathsf{\Omega }}(X)=\left\{\begin{array}{c}0,\hspace{0.33em}X\in \mathsf{\Omega }\\ \infty ,\hspace{0.33em}\mathrm{otherwise}\hspace{0.33em}\end{array}\right.$,

 $L$ and $\mathsf{\Lambda }$ are the Lagrangian multipliers for variables $Z$ and $U{V}_{}^H$ respectively. The regularization parameter $\beta$ is used to balance the low rankness and sparsity constraints (i.e. TV), the penalty parameters ${\rho }_1>0$ and ${\rho }_2>0$ generally affect the convergence of the algorithm. By applying ADMM, each sub-problem is performed at each iteration t as follows:

$$X_{}^t=\arg \underset{X}{\min }{\tau }_{\mathsf{\Omega }}(X)+\frac{{\rho }_1}{2}{‖\mathsf{{\rm M}}{\mathsf{\Phi }}_{1}X-U_{}^{t-1}{V}_{}^{{{}^{(t-1)}}^H}+{\mathsf{\Lambda }}_{}^{{}^{t-1}}‖}_F^2$$

(5)

$$U_{}^t=\arg \underset{U_{}}{\min }{‖U_{}‖}_F^2+\frac{{\rho }_1}{2}{‖\mathsf{{\rm M}}{\mathsf{\Phi }}_1{X}^t-U{V}_{}^{(t-1)H}+{\mathsf{\Lambda }}_{}^{t-1}‖}_F^2$$

(6)

$$V_{}^t=\arg \underset{V_{}}{\min }{‖V_{}‖}_F^2+\frac{{\rho }_1}{2}{‖\mathsf{{\rm M}}{\mathsf{\Phi }}_1{X}^t-U_{}^t{V}_{}^H+{\mathsf{\Lambda }}_{}^{t-1}‖}_F^2$$

(7)

$$Z^t=\arg \underset{Z}{\min }{‖Z‖}_1+\frac{{\rho }_2}{2}{‖D{X}^t-Z+L^{t-1}‖}_F^2$$

(8)

$${\mathsf{\Lambda }}_{}^t={\mathsf{\Lambda }}_{}^{t-1}+\mathsf{{\rm M}}{\mathsf{\Phi }}_1{X}^t-U_{}^t{V}_{}^{(t)H}$$

(9)

$$L^t=L^{t-1}+D{X}^t-Z^t$$

(10)

The initial $U$ and $V$ can be determined by solving the following optimization problem using the LMaFit method [43].

$$\underset{U_{},V_{},X}{\min }{‖U_{}{V}_{}^H-\mathsf{{\rm M}}{\mathsf{\Phi }}_{1}X‖}_F^2\hspace{0.33em}\hspace{0.33em}subject\hspace{0.33em}to\hspace{1em}X(\mathrm{i},\hspace{0.33em}\mathrm{j})=Y(\mathrm{i},\hspace{0.33em}\mathrm{j}),\hspace{0.33em}\forall (\mathrm{i},\hspace{0.33em}\mathrm{j})\in \mathsf{\Omega }$$

(11)

The whole algorithm for solving the model (3) is shown in Table 2.

3.3. Method 2: The Low Tubal-Rank-Based Method

Tensor-SVD decomposition has been efficiently used in the video image completion and dynamic MR image reconstruction problem [16,44,45,46]. Since the color image is highly unbalanced in the size of three dimensions, which is not suitable for the low tubal rank constraint, we exploit the second flexible QA scheme to deeply dig out the potential low tubal-rank prior information.

Considering that tubal rank minimizations are more efficient for the balanced tensor [10], we first turn the unbalanced image into the balanced order-3 data by the second flexible QA scheme.

Take the color image size of 256×256×3 as an example, as shown in Figure 6 (a), we can obtain the order-4 tensor size of 128×128×4×3 by the basic QA schemes, and then multiplying the basic QA schemes we can obtain the order-4 tensor size of 64×64×4×4×3. Lastly, we reshape the order-4 tensor into the balanced order-3 tensor size of 64×64×48. Here, the context of ‘balanced’ is that the size changes from the unbalanced 256×256×3 to the more balanced size of 64×64×48. In practice, the size of the designed order-3 tensor should be as balanced as possible. We call the above the second flexible QA scheme.

In Figure 6 (b), we show the low tubal rankness of the balanced order-3 data (with the size of $n_1\times n_2\times n_3=64\times 64\times 48$ here) by plotting ${\delta }_j$ which is defined as follows.

$${\delta }_j=\frac{1}{n_3}{\displaystyle \sum _{i=1}^{n_3}T(i,\hspace{0.33em}i,\hspace{0.33em}j)},\hspace{1em}i=1,\hspace{0.33em}2,\hspace{0.33em}\cdots ,\hspace{0.33em}\min (n_1,\hspace{0.33em}{n}_2)$$

Then, TNN is used to enforce the tensor tubal rank in the image inpainting model as follows.

$$\underset{X}{\min }{‖{\mathsf{\Phi }}_{2}X‖}_{TNN}\hspace{1em}\mathrm{s}ubject\hspace{0.33em}to\hspace{0.33em}\hspace{0.33em}X(\mathrm{i},\hspace{0.33em}\mathrm{j})=Y(\mathrm{i},\hspace{0.33em}\mathrm{j}),\hspace{0.33em}\forall (\mathrm{i},\hspace{0.33em}\mathrm{j})\in \mathsf{\Omega }$$

(12)

where ${\mathsf{\Phi }}_2$ denotes the operator of permuting the color image into a more ‘balanced’ order-3 tensor by the second flexible QA scheme. Combining the low tubal rank and sparsity, we introduce auxiliary variables $\mathcal{B}={\mathsf{\Phi }}_{2}X$, and $Z=DX$, then rewrite (12) as the following unconstrained convex optimization problem.

$$\begin{array}{l}\underset{X}{\min }{\tau }_{\mathsf{\Omega }}(X)+{{\displaystyle \sum _i^{I_3}‖{\overline{\mathcal{Z}}}^{(i)}‖}}_{*}+\frac{\rho }{2}{‖{\mathsf{\Phi }}_{2}X-\mathcal{Z}+\mathsf{\Lambda }‖}_F^2\\ \hspace{1em}\hspace{0.33em}\hspace{0.33em}+\frac{\beta }{2}{‖Z‖}_1+\frac{\beta {\rho }_2}{2}{‖DX-Z+L‖}_F^2\end{array}$$

(13)

where $I_3$ is the third size of the 3-order tensor ${\mathsf{\Phi }}_{2}X$. We conduct the algorithm by ADMM for solving model (13) as shown in Table 3.

3.4. Method 3: The Low TT-Rank-Based Method

TT decomposition works better on higher-order tensors than Tucker decomposition. To fulfill TT decomposition efficiently, we first exploit the third flexible QA scheme to permute the 3-order image into a higher-order tensor. Based on the basic QA scheme, high-order tensors can be obtained flexibly.

The third flexible QA scheme is shown below. We take a $16\times 16$ matrix as an example, as shown in Figure 7 (a). We first turn a 2-order matrix into a 3D tensor via the basic QA scheme and then repeat the basic QA to obtain the final 4D tensor with the size of $4\times 4\times 4\times 4$. The entry comes from the i^th smaller matrix of the first basic QA scheme, and the j^th smaller matrix of the second basic QA scheme is labeled as the MATLAB notation $(:,:,i,j)$. By analogy, the third flexible QA scheme can permute a matrix with the size of $4^P\times 4^Q$ to order- $\min \{P,Q\}$ tensor with the size of $\underset{\min \{P,Q\}}{\underbrace{4\times 4\times \cdots \times 4}}$. An RGB image with the size of $4^P\times 4^Q\times 3$ can be permuted into an order-$\min \{P,Q\}+1$ tensor with the size of $\underset{1+\min \{P,Q\}}{\underbrace{4\times 4\times \cdots \times 4\times 3}}$. The third flexible QA scheme should ensure that the designed tensor has a higher order.

We permute the Lena image into a high-order tensor via the third flexible QA scheme, and then obtain the TT matrices of the augmented tensor. We name those TT matrices as QA-TT matrices, and their singular values are shown in Figure 7 (b), which demonstrates the low TT rankness of the rearranged tensor.

Then, we enforce the low TT rankness to improve the inpainting accuracy. The third model is as follows.

$$\underset{X}{\min }{\displaystyle \sum _{\mathrm{n}=1}^N{\alpha }_{\mathrm{n}}{‖{\mathcal{T}}_{\mathrm{n}}{\mathsf{\Phi }}_{3}X‖}_{*}}\hspace{1em}\mathrm{s}ubject\hspace{0.33em}to\hspace{0.33em}\hspace{0.33em}X(\mathrm{i},\hspace{0.33em}\mathrm{j})=Y(\mathrm{i},\hspace{0.33em}\mathrm{j}),\hspace{0.33em}\forall (\mathrm{i},\hspace{0.33em}\mathrm{j})\in \mathsf{\Omega }$$

(14)

where ${\mathsf{\Phi }}_3$ stands for the third flexible QA used to permute image $X$ into a high-dimensional tensor. We name the tensor obtained by the third flexible QA scheme as a QA tensor. ${\mathcal{T}}_n$ is the operator that converts a tensor into the n^th TT matrix, $n=1,\hspace{0.33em}2,\hspace{0.33em}\cdots ,\hspace{0.33em}N$. The order of QA tensor is $N$. The inverse operators corresponding to $\mathsf{\Phi }$ and ${\mathcal{T}}_n$ are ${\mathsf{\Phi }}^{-1}$ and ${\mathcal{T}}_n{}^{-1}$ respectively. The weight ${\alpha }_n$ is given by:

$${\alpha }_n=\frac{{\theta }_n}{{\displaystyle {\sum }_{n=1}^{N-1}{\theta }_n}}\mathrm{with} {\theta }_n=\min ({\displaystyle \prod _{l=1}^n{I}_l},{\displaystyle \prod _{l=k+1}^N{I}_l})$$

(15)

where $I_1\times I_2\times \cdots \times I_N$ is the size of the QA tensor.

Combining the low TT rank and sparsity constraints, we introduce auxiliary variables $\mathrm{Z}=DX$ and $U_n{V}_n^H={\mathcal{T}}_n{\mathsf{\Phi }}_{3}X$, rewrite (14) as the following unconstrained convex optimization problem, for all $n=1,\cdots ,N-1$.

$$\begin{array}{l}\underset{U_n,V_n,{\mathsf{\Lambda }}_n,L,Z,X}{\min }{\tau }_{\mathsf{\Omega }}(X)+\frac{1}{2}{\displaystyle \sum _{n=1}^{N-1}{\alpha }_n({‖U_n‖}_F^2+{‖V_n‖}_F^2)}+\frac{\beta }{2}{‖Z‖}_1\\ +\frac{{\rho }_1}{2}{\displaystyle \sum _{n=1}^{N-1}{\alpha }_n{‖{\mathcal{T}}_n{\mathsf{\Phi }}_{3}X-U_n{V}_n^H+{\mathsf{\Lambda }}_n‖}_F^2)}+\frac{\beta {\rho }_2}{2}{‖DX-Z+L‖}_F^2\end{array}$$

(16)

By applying ADMM, each sub-problem is performed at each iteration t. Lastly, we obtain $X$ by $X^{*}={\displaystyle {\sum }_{n=1}^{N-1}{\alpha }_n}{X}_n^{\ast }$, where $X_n^{\ast }$ represents the optimal solution of the nth subproblem. The whole algorithm for solving the model (16) is shown in Table 4.

4. Experimental Results and Analyses

In this section, we conduct the above methods 1-3 for solving image inpainting problems. For simplicity, we denote methods 1-3 which only exploit low unfolding matrix rank, low tensor tubal rank, and low tensor train rankness as UfoldingLR, TTLR, and tSVDLR methods respectively. The methods that enhance the low rank and total variation constraints simultaneously are denoted as UnfoldingLRTV, tSVDLRTV, and TTLRTV methods respectively. We denote the low matrix-rank completion method which is solved by the model (17) and ADMM algorithm as the MatrixLR method.

$$\underset{X}{\min }\hspace{1em}rank(X)\hspace{1em}subject\hspace{0.33em}to\hspace{1em}X(\mathrm{i},\hspace{0.33em}\mathrm{j})=Y(\mathrm{i},\hspace{0.33em}\mathrm{j}),\hspace{0.33em}\forall (\mathrm{i},\hspace{0.33em}\mathrm{j})\in \mathsf{\Omega }$$

(17)

We denote the method that only exploits sparsity in the gradient domain and is solved by the ADMM algorithm as the TV method. Besides, we conduct the following numerous close methods for comparison, some of their codes are available online.

STDC: the method exploited the images into three factor matrices and one core tensor for image inpainting [7,19,20]2.

HaLRTC: the method constrained the low rankness of the three mode-n matrices caused by decomposition of a color image for inpainting and which was solved by the ADMM [21,22]3.

SPCTV4: the smooth PARAFAC tensor completion and total variation method [23], which used the PD (PARAFAC decomposition, a derivation of Tucker decomposition) framework and constrained the TV on every factor matrix of PD respectively.

LRTV: the methods combined the constraints of the low rankness of every mode-n matrix and the TV regularization on every mode-n matrix for color image inpainting [24,25]5.

FBCP: the inpainting methods based on Bayesian tensor completion [26,27,28,29]6.

All simulations were carried out on Windows 10 and MATLAB R2019a running on a PC with an Intel Core i7 CPU 2.8GHz and 16GB of memory. For a fair comparison, every method is conducted with its optimal parameters to ensure every method has the best performance. The reconstruction quality is quantified using the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM)7 [37]. The original color images (from the standard image database) and missing patterns used in the experiments are shown in Figure 8.

We set the maximum number of iterations $t_{\max }=100$ and convergence condition ${\eta }_{tol}={10}^{-6}$ in all our methods (UnfoldingLRTV, tSVDLRTV, and TTLRTV). The pixel range of all the images is normalized to 0-1. In UnfoldingLR, tSVDLR, and TTLR methods, we set $\hspace{0.17em}{\rho }_1\hspace{0.17em}=$ 0.04, 0.002, and 0.6 respectively. In UnfoldingLRTV, Tsvdlrtv, and TTLRTV methods, we set the parameter set $({\rho }_1\hspace{0.17em},\hspace{0.33em}\beta ,\hspace{0.17em}\hspace{0.33em}{\rho }_2)$ as (0.4, 0.004, 2), (0.6, 0.07, 0.1), and (0.7, 0.03, 0.1) respectively.

4.1. Analyses of the Three Flexible QA Schemes

Next, we call the first, second, and third flexible QA schemes QA scheme briefly. The PSNRs (dB)/SSIMs of the UnfoldingLR, tSVDLR, and TTLR methods with and without the QA scheme are shown in Table 5. The red numerical values correspond to the worst results. We can see that, without the QA scheme, Lena and Airplane cannot be recovered. The UnfoldingLR, tSVDLR, and TTLR methods with the QA scheme have better numerical results than those without the QA scheme. In the low matrix-rank completion method (i.e. MatrixLR), no QA scheme is applied, i.e. the color image is dealt with as three-channel matrices directly.

Due to the support of the QA scheme, the low tensor-rank based methods (TTLR, tSVDLR, and UnfoldingLR) with the QA scheme provide better results than the traditional low matrix-rank completion method (i.e. MatrixLR method). So, the QA scheme is successful to be used as the first step to deeply explore the low tensor rank prior to an image.

The KA scheme and the third flexible QA scheme both can rearrange an image into a high-order tensor. However, our QA scheme is different from the KA scheme used in [21]. The KA scheme maintains the local block similarity of the image, while the third flexible QA scheme uses adjacent pixels to maintain the global similarity of the image. We conduct the comparison of KA and the third flexible QA scheme under the corresponding TMac-TTKA [21] and TTLR methods. As shown in Figure 9, the small blocks are obvious in the recovered images by the TMac-TTKA method. The images recovered by the TTLR method preserve more details and without the obvious blocks.

4.2. Analyses of the Methods Exploiting Both Low Rankness and Sparsity

In this section, we analyze the recovery results of the methods both exploiting low rankness and sparsity. Figure 10, Figure 11 and Figure 12 show the visual comparisons of the eleven methods for recovering the House, Lena, and Baboon images respectively. Table 6 shows the PSNR (dB)/SSIM results of the nine methods for recovering different color images under different missing patterns. Figure 13 depicts the PSNR curves of the inpainting results of the different methods, the missing ratio ranges from 10% to 70% under a random missing pattern.

As shown in Figure 10, Figure 11, Figure 12 and Figure 13 and Table 6, compared to the numerous close STDC, HaLRTC, FBCP, TMac-TTKA, SPCTV, and LRTV methods, the UnfoldingLRTV, tSVDLRTV, and TTLRTV methods have the super performance on both visual and quantity results. The SPCTV and LRTV methods also enhance the low rankness and sparsity simultaneously, but the results are worse than our methods.

Table 7 shows the PSNR (dB)/SSIM results of the eight methods: MatrixLR method only constrains the low matrix rank; TV method only exploits the TV prior; The UnfoldingLR, tSVDLR, and TTLR methods only constrain the low unfolding matrix rank, low tubal rank and low TT rank respectively; The UnfoldingLRTV, tSVDLRTV, and TTLRTV methods combine both sparsity and low tensor rankness. As shown in Table 7, the combination of sparsity and low tensor rankness constraints can yield better inpainting results than enforcing sparsity or low rankness alone. TTLR method is more efficient than the MatrixLR, and TV methods. The results of the tSVDLR method and TTLR method are comparable. The UnfoldingLR method provides the best results among the TTLR, tSVDLR, TuckerLR, MatrixLR, and TV methods. UnfoldingLRTV, tSVDLRTV, and TTLRTV methods have improved numerical results than the corresponding UnfoldingLR, tSVDLR, and TTLR methods, which demonstrates that TV prior is efficient in improving the accuracy of low-rank based inpainting methods.

The visual and numerical PSNR (dB)/SSIM comparisons of our methods for recovering the pepper image under 80% random missing patterns are shown in Figure 14. In the first row of Figure 14, the methods only exploit low-rank constraints. As shown in the color box, there are small blocky errors in the recovered image, these are caused by the QA scheme. This phenomenon can be solved by combining the constraints of low rank and sparsity (TV), as shown in the second row of Figure 14.

All in all, due to the support of the QA scheme and the efficient TV prior, the low tensor-rank based methods (UnfoldingLRTV, tSVDLRTV, and TTLRTV) are superior to other close low tensor-rank based methods. The UnfoldingLRTV method provides the best results among all the methods conducted in this paper.

4.3. Analyses of TTLR and TTLRTV Methods

In this section, we mainly focus on the analyses of the TT based methods (i.e. TTLR and TTLRTV) in detail. Since TT rank is multi-rank, how does every TT matrix rank affect the final result? We answer this question with the below experimental results.

We conducted the experiments on recovering House, Lena, and Airplane images with a size of 256×256×3. The random missing patterns have four missing ratios: 10%, 30%, 50%, and 70% respectively. We label the 8 TT matrices as k=1, 2, …, 8. Then the PSNR (dB) results of $X_n^{*}$ (the optimal solution of the n^th subproblem which exploits the n^th TT matrix rank) in TTLR and TTLRTV methods are shown in Figure 15.

From Figure 15, we can see that, the PSNR (dB) results of each subproblem is steeply different in the TTLR method, which demonstrates that each TT matrix rank contributes different PSNR result. Since there are no rules to find which TT matrix rank meets the best PSNR result, we should combine each solution of $X_n^{*}$ to obtain the final $X^{*}$, i.e. $X^{*}={\displaystyle {\sum }_{n=1}^{N-1}{\alpha }_n{X}_n^{*}}$. Comparing the PSNR curves of TTLR and TTLRTV method in Figure 15, the PSNR (dB) results of each subproblem is slightly different in the TTLRTV method which demonstrates that the combination of TT and TV can make the PSNR more balanced among all k.

4.4. Runtime and Complexity Analysis

From a high-dimensional curse perspective, converting an image to a higher-order tensor can result in increased complexity, which inevitably leads to a longer runtime. We compare our methods (UnfoldingLRTV, tSVDLRTV, and TTLRTV) with the traditional MatrixLR method and the close STDC, HaLRTC, FBCP, SPCQV, and LRTV methods in running time, as shown in Table 8.

UnfoldingLRTV methods: In the first step, the QA scheme is used to decompose a single image into several small graphs. Because of the similarity of these small graphs, the QA tensor can be reduced to a matrix with a low-rank structure in an unfolding way. Ignoring TV constraints, the unfoldingLRTV method only needs to solve the low-rank matrix completion problem of an unfolding matrix, so the running time is similar to the traditional MatrixLR method, and the accuracy is higher than the traditional MatrixLR method.

The tSVDLRTV methods: Since the color image is highly unbalanced in the size of three dimensions, which is not suitable for the low tubal rank constraint, we use the QA scheme to rearrange an image into a third-order tensor with a more balanced size of every dimension. Then we use TNN to constrain the low tubal rank of the rearranged tensor, due to the fast Fourier scheme, it is necessary to perform a low-rank matrix constraint on each frontal slice after the third-dimensional Fourier transform. At this time, the SVD decomposition process will increase the time consumption.

TTLRTV methods: TT multi-rank is the combination of the rank of each TT matrix. The TTLRTV method essentially completes the same data amount N-1 times, where N is the order of the QA tensor. So, although the TITRTV method is effective, it is necessarily more computationally expensive than the low matrix-rank completion method.

In summary, among the three methods, the UnfoldingLRTV method achieves the best performance both in accuracy and runtime; The TTLRTV method reaches better accuracy, but it is time-consuming; The tSVDLRTV method has moderate performance both in runtime.

All in all, the above three methods can deeply exploit the potential low-rank prior of an image and have been successfully used for image inpainting problems, which demonstrates that the three flexible QA schemes are perfect ways to explore the low-rank prior of an image.

5. Conclusions

To effectively explore the potential of low tensor rank prior to an image, we first exploited a rearrangement scheme (QA) for permuting the color image (3-order) into three flexible rearrangement forms (with more efficient low tensor rank structure). Based on the scheme, three optimization models by exploiting the low unfolding matrix rank, low tensor tubal rank, and low TT multi-rank were proposed to improve the accuracy in image inpainting. Combined with TV constraints, we developed efficient ADMM algorithms for solving those three optimization models. The experimental results demonstrate that our low tensor-rank-based methods are effective for image inpainting, and are superior to the low matrix-rank completion method and numerous close methods. The low tensor rank constraint is effective for image inpainting, which is mainly due to the support of the QA scheme.