PEIPNet: Parametric Efficient Image-Inpainting Network with Depthwise and Pointwise Convolution

1. Introduction

Image inpainting attempts to generate masked regions with visually satisfying image structures and regional features. This task has long been a major research area in computer vision. However, rapid changes have occurred in recent years with the emergence of deep learning. For example, traditional methods [1,2,3,4,5,6,7,8,9] only used known pixels for weighted replication or diffusion. However, deep learning-based approaches [10,11,12,13,14,15,16,17,18] continuously compress masked images into a dense latent space and fill-in the missing pixels by restoring high-level semantic information. The strength of deep learning-based methods is clearly demonstrated in large-hole inpainting tasks.

Various deep learning-based models have been introduced with the development of convolutional neural networks (CNNs) and generative adversarial networks (GANs) [19]. For instance, in accordance with the traditional concept, the methods [16,17,20,21,22] integrate a patch-matching algorithm into the latent space to effectively generate pixels and produce results that are visually pleasing. To address the limitations of vanilla convolution, partial convolution (PConv) and gated convolution (GConv) were proposed in [13,23], in which valid pixels were conditioned by the mask that acted as a prior. The framework of inpainting models also plays a key role in performance enhancement. Specifically, the methods [24,25,26,27] adopted a two-stage inpainting architecture in which the overall structural features of the masked regions were forecast in the first stage, while the final result was predicted in the second stage by adopting the output of the previous step as a constraint. Typically, the Edge Connect (EC) [24] recovers the missing edges in the first stage by piling up numerous residual blocks with dilated convolutional layers, which increases the receptive field of the model and extracts global features. In the second stage, the restored edge map is employed in texture synthesis and ensures the visual unity of the final result. However, these approaches often require separate training, in which the end-to-end training strategy cannot be applied and increases the training complexity. Moreover, because two distinct models are employed to form a generator, the number of parameters and model complexity also increase as the probability of having an overfitting problem increases. This increases the usage of the operational memory, which hinders the use of inpainting models on restricted computational resources.

To solve these problems, we proposed the Parametric Efficient Image InPainting Network (PEIPNet) with a one-stage inpainting framework. First, we adopted depthwise and pointwise convolutions instead of the vanilla convolution to minimize the number of model parameters, which was inspired by [28]. Second, we employed spatially-adaptive denormalization (SPADE) [29] to conditionally normalize the feature maps according to the mask. Third, we introduced a dense dilated convolution module (DDCM), inspired by [30], to gradually fill-in the missing regions at different scales and mitigate the gradient-vanishing problem. Finally, efficient self-attention (ESA) [31] was utilized to capture long-range information from unmasked areas to enhance the inpainting accuracy.

To the best of our knowledge, the proposed PEIPNet is the first method in the image-inpainting task that has less than one million model parameters but ensures high performance. The contributions of this study are summarized as follows:

• An efficient one-stage inpainting framework with effective modules is proposed to reduce the number of model parameters and computational costs to mitigate the overfitting problem when trained on small datasets, with potential applications in various environments.
• Qualitative and quantitative evaluations conducted on different public datasets demonstrate the excellent performance of our method compared with state-of-the-art models.

2. Related Works

2.1. Image Inpainting by Patch-Based Methods

Patch-based methods were first introduced for texture synthesis [1,3], and then adopted in [32] for image inpainting to fill-in masked areas at the image level. These approaches normally inspect and copy comparable patches from a database or uncontaminated background into missing regions according to the distance metrics between different patches, i.e., Euclidean distance and SIFT distance [33]. Bertalmio et al. [4] merged patch-based texture combination techniques with diffusion-based dispersion under image decomposition. PatchMatch [9] was proposed to search for similar matches between image patches. Therefore, patch-based designs for image inpainting can produce sharp outputs with similar contexts. However, generating semantically plausible images by a patch-based approach remains difficult because a high-level understanding of the images is required.

2.2. Image Inpainting by Deep Generative Methods

Generative models based on neural networks for image inpainting typically encode a damaged image into a latent feature. In the latent space, the masked areas are filled-in at the feature level, and the feature maps are decoded and restored into an image. In recent years, studies based on deep generative models have yielded promising results. The Context Encoder (CE) [10] is one of the first neural networks that adopted deep feature learning and adversarial training [19]. The CE can generate visually appealing outputs for semantic hole-filling. In terms of the loss function, a guidance loss was proposed in [16], which forced the feature maps produced in the decoder to become similar to those of the ground-truth images made in the encoder. The methods in [11,24] employed dilated convolution to increase the receptive field of the model and capture global features. A two-stage inpainting framework was utilized in [24,25,26,27] to fill-in the pixels with the constraints generated in the first stage. Instead of vanilla convolution, PConv [14] and GConv [23] were designed to eliminate the effects that are induced by placeholder values in the masked areas of an image. A contextual attention (CA) layer [18] was proposed to fill-in the lost pixels with similar patches from undistorted areas in high-level feature maps. With regard to the CA layer, Sagong et al. [21] introduced a novel parallel extended-decoder path with a modified CA module to reduce the computational cost. However, minimizing the capacity of the model to increase its efficiency and solve the overfitting problem remains a challenge.

2.3. Conditional Normalization

A normalization layer for feature extraction is commonly applied in deep neural networks to aid the training process.

For instance, batch normalization (BN) [34] normalizes the activation maps across batch and spatial dimensions that affect generative networks. Instance normalization (IN) [35] differs from BN because it normalizes the features only across spatial dimensions and enhances the outcomes of numerous generative tasks, such as style transformation. Layer normalization (LN) [36] normalizes activations across the channel and spatial dimensions, and helps train recurrent neural networks more stably. Group normalization (GN) [37] normalizes the features of grouped channels of an instance, which boosts the performance of specific vision tasks such as object detection.

Conditional normalization differs from a single set of affine parameters in the aforementioned normalization approaches. Conditional normalization methods typically utilize external information to learn multiple sets of affine parameters. Conditional IN [38], adaptive IN [39], conditional BN [40], and SPADE [29] were introduced in image synthesis tasks. In the image-inpainting task, region normalization (RN) [41] was introduced for spatial region-wise normalization to overcome the limitations of typical feature normalization methods without considering the impact of the corrupted areas of the input image during the normalization process.

2.4. Neural Networks with Lightweight Architecture

Lightweight neural networks have been introduced to allow model developers to select a small network that corresponds to resource restrictions, i.e., latency and size. In particular, many studies have been conducted on image classification tasks to reduce the model capacity. Depthwise separable convolution was adopted in [42] and MobileNets [28] to reduce the computations in the first few layers. In contrast, flattened networks [43] construct a network with fully factorized convolution and demonstrate the effectiveness of intensely factored models. Factorized convolution was employed in factorized networks [44] similar to that in topological conjunctions. Afterwards, Xception [45] scaled-up the depthwise separable convolution to exceed the performance of Inception-V3 [46]. SqueezeNet [47] is another remarkably small network that uses a bottleneck method to reduce the number of required parameters. Structured transform networks [48] and deep fried convnets [49] were proposed to reduce computational costs. However, none of these strategies have been applied in image-inpainting tasks to scale down the model capacity.

3. Methodology

In this section, we first introduce the architecture of the proposed method and then explain the various loss functions used for training.

3.1. Generator Architecture

Figure 1 shows the architecture of the proposed method, which employs an autoencoder framework. To minimize the number of model parameters, we introduced a combination of depthwise and pointwise convolutions, as in [28]. Depthwise and pointwise convolutions were employed sequentially at the encoder, and applied in the opposite order at the decoder. In both parts, depthwise convolutions with a kernel size of $3\times 3$ were used to extract features and increase the receptive field. Pointwise convolutions with a kernel size of $1\times 1$ were used to alter the number of channels. For all convolutional modules, spectral normalization [50] was used to stabilize the training process. SPADE [29] was chosen to conditionally normalize the masked and unmasked regions according to the semantic prior information. The SPADE framework is comprehensively analyzed in Section 3.1.1. Then, the LeakyReLU (LReLU) is used as the activation function with its hyperparameter set to 0.2.

Given a masked input image with a size of $256\times 256\times 3$, the encoder first expanded the number of channels to 48. The spatial size of the feature map was then downsampled by half using a depthwise convolutional module, while the number of channels was doubled to extract the feature map of size $128\times 128\times 96$. The encoder again applied the same operation, which induced the feature map to become $64\times 64\times 192$. At this point, the masked regions were still not filled with appropriate pixels. Hence, we proposed a DDCM to fill-in the missing areas and aggregate features with different receptive fields. The DDCM structure is thoroughly discussed in Section 3.1.2.

After passing through the encoder, the decoder was used to recover the resolution and generate the final output. The decoder first upsampled the input feature map using the nearest-neighbor interpolation with a scaling factor of two. To obtain the spatial information, the output was concatenated channel-wise with the feature map from the encoder, which has the same spatial size. Because the masked regions in the feature map from the encoder were not filled, we applied the DDCM to the feature map right before concatenation. Moreover, we employed ESA [31] to fully extract useful nonlocal information from the aggregated feature map. The design of ESA is described in Section 3.1.3. After ESA, the number of channels in the feature map was reduced from 288 to 96 using the pointwise and depthwise convolutional modules, which yielded the feature map of size $128\times 128\times 96$. The decoder used the same procedure to extract the feature map of size $256\times 256\times 48$. Finally, a pointwise convolutional module was employed to match the number of channels with the original input and produce the final result of size $256\times 256\times 3$.

3.1.1. Spatially-Adaptive Denormalization

SPADE [29] was introduced as a conditional normalization method for image-to-image translation tasks. SPADE uses a semantic segmentation mask as a prior to learn the mapping function that can transform an input segmentation mask into a realistic image.

The SPADE employed in our model used a binary mask $\mathbf{M}$ as the prior. Let $\mathbf{M}\in {\mathbb{L}}^{H\times W}$, where $\mathbb{L}$ is a set of integers $[0,1]$ that denote the unmasked and masked regions, and H and W are the mask height and width, respectively. Assuming the features of the i-th layer of a CNN with a batch of N samples as ${\mathbf{f}}^i$, the number of channels in the layer as $C^i$, the height and width of the feature map in the layer as $H^i$ and $W^i$, respectively, the feature value at site $(n\in N,c\in C^i,y\in H^i,x\in W^i)$ is formulated as

$${\gamma }_{c,y,x}^i\left([\mathbf{M},{\mathbf{M}}^{\prime }]\right)\odot \frac{{\mathbf{f}}_{n,c,y,x}^i-{\mu }_c^i}{{\sigma }_c^i}\oplus {\beta }_{c,y,x}^i\left([\mathbf{M},{\mathbf{M}}^{\prime }]\right),$$

(1)

where $[\mathbf{M},{\mathbf{M}}^{\prime }]$ are the set of masks in which ${\mathbf{M}}^{\prime }$ denotes the inversion of binary mask $\mathbf{M}$; ${\mathbf{f}}_{n,c,y,x}^i$ is the feature at the site before normalization; ${\mu }_c^i$ and ${\sigma }_c^i$ are the mean and standard deviation of the features in channel c, respectively; and ⊙ and ⊕ are the element-wise multiplication and addition, respectively. Nonparametric BN [34] was employed to compute ${\mu }_c^i$ and ${\sigma }_c^i$, which are expressed as

$${\mu }_c^i=\frac{1}{N{H}^i{W}^i}\sum _{n,y,x}{\mathbf{f}}_{n,c,y,x}^i$$

(2)

$${\sigma }_c^i=\sqrt{\frac{1}{N{H}^i{W}^i}{\sum }_{n,y,x}{({\mathbf{f}}_{n,c,y,x}^i-{\mu }_c^i)}^2}.$$

(3)

The parametric variables ${\gamma }_{c,y,x}^i\left([\mathbf{M},{\mathbf{M}}^{\prime }]\right)$, and ${\beta }_{c,y,x}^i\left([\mathbf{M},{\mathbf{M}}^{\prime }]\right)$ in Equation 1 are the learned modulation parameters of the normalization layer. Compared with standard normalization layers, such as BN, the SPADE is well suited for the image-inpainting task because the modulation parameters adapt to the binary mask where masked and unmasked regions are given as the prior. A structural overview of the SPADE is shown in Figure 2.

3.1.2. Dense Dilated Convolution Module

We propose the DDCM to fill-in the missing regions by combining various feature maps with different receptive fields in each stage. The main components of the DDCM are dense blocks [30] and dilated convolutional modules. We first explain the structure of the proposed DDCM and then present the reasons for suitability in image-inpainting tasks.

Given the input feature map $F_{DDCM}$ of size $H\times W\times C$ that passes through depthwise and pointwise convolutional modules. To reduce the number of model parameters, a bottleneck layer with a pointwise convolutional layer reduces the number of channels to $C^{\prime }$ to generate the feature map $F_{DDCM}^0$, which is expressed as

$$F_{DDCM}^0=BL\left(PC\left(DC\left(F_{DDCM}\right)\right)\right),$$

(4)

where $C^{\prime }=\frac{C}{2}$, and $DC$, $PC$, and $BL$ denote the depthwise and pointwise convolutional module and bottleneck layer, respectively.

Then, a depthwise convolutional module with a dilation rate of two and a pointwise convolutional module were applied to obtain the feature map $F_{DDCM}^1$, which is formulated as

$$F_{DDCM}^1=PC\left(DD{C}_{r=2}\left(F_{DDCM}^0\right)\right),$$

(5)

where $DDC$ represents the dilated depthwise convolutional module with the subscript r denoting the dilation rate.

The same operation with a dilation rate of four was employed to extract the feature map $F_{DDCM}^2$, which is defined as

$$F_{DDCM}^2=PC\left(DD{C}_{r=4}\left(F_{DDCM}^1\right)\right).$$

(6)

Next, two feature maps, $F_{DDCM}^1$ and $F_{DDCM}^2$, were concatenated channel-wise to produce $[F_{DDCM}^1,F_{DDCM}^2]$ for feature aggregation. The combined feature map then went through the bottleneck layer to decrease the number of channels from $2C^{\prime }$ to $C^{\prime }$. A depthwise convolutional module with a dilation rate of six and a pointwise convolutional module were applied to construct the feature map $F_{DDCM}^3$, which is expressed as

$$F_{DDCM}^3=PC\left(DD{C}_{r=6}\left(BL\left([F_{DDCM}^1,F_{DDCM}^2]\right)\right)\right).$$

(7)

Using all the previous features, the same operation as in Equation 7 was applied to yield the feature maps $F_{DDCM}^4$ and $F_{DDCM}^5$, which are formulated as

$$F_{DDCM}^4=PC\left(DD{C}_{r=8}\left(BL\left([F_{DDCM}^1,F_{DDCM}^2,F_{DDCM}^3]\right)\right)\right)$$

(8)

$$F_{DDCM}^5=PC\left(DD{C}_{r=10}\left(BL\left([F_{DDCM}^1,F_{DDCM}^2,F_{DDCM}^3,F_{DDCM}^4]\right)\right)\right).$$

(9)

Finally, all the extracted feature maps were combined into one feature map to utilize the effective information. This was then fed into the bottleneck layer to reduce the number of channels from $5C^{\prime }$ to C. Depthwise and pointwise convolutional modules were applied to produce the final output $F_{DDCM}^{{}^{\prime }}$ of size $H\times W\times C$, which is defined as

$$F_{DDCM}^{{}^{\prime }}=PC\left(DC\left(BL\left([F_{DDCM}^1,F_{DDCM}^2,F_{DDCM}^3,F_{DDCM}^4,F_{DDCM}^5]\right)\right)\right).$$

(10)

As the feature map was fed forward, the a dilation rate of the depthwise convolutional module increased by two. Increasing the dilation rate expanded the effective receptive field, which enabled the model to look over much larger areas of the feature map. This process is applicable to image-inpainting tasks because the network can fill-in the masked regions using various types of information from the entire feature map. A structural overview of the DDCM is shown in Figure 3.

The dense block was first introduced in [30] to mitigate the vanishing-gradient problem, enhance feature propagation, and stimulate the reuse of the feature map. For the DDCM, we employed the dense block for two main reasons.

The first reason is to help the model converge to a better minimum point. Because the DDCMs are located in the middle of the proposed method and skip the connections between the encoder and decoder, the gradient may vanish because of the deeply stacked convolutional modules. Hence, adding a dense block provides various paths to pass the gradient at the current location to the input, which eventually solves the gradient-vanishing problem.

The second reason is to gradually fill-in the missing areas in the feature map with the relevant pixels. Although the feature maps were extracted and downsampled to different scales, the missing areas were not provided with the appropriate values. Thus, the dense block was adopted to progressively generate pixels while inducing minimal artifacts. The feature maps at the front part of the DDCM (i.e., $F_{DDCM}^1$ and $F_{DDCM}^2$) were extracted using local information. Therefore, passing feature maps at every latter stage acted as the prior information, which helped the model continuously draw out features from local to global regions with an increased receptive field.

Thus, the DDCM is suitable for image-inpainting tasks for the aforementioned reasons. The effect of the DDCM is thoroughly analyzed in Section 4.7.

3.1.3. Efficient Self-Attention

For our model, ESA [31] was employed to extract nonlocal information while retaining the minimum number of model parameters. In this section, we explain the details of ESA by comparing it with dot-product self-attention.

Dot-product self-attention [51] is a method to model long-range interactions in neural networks. Let ${\mathbf{f}}^i\in {\mathbb{R}}^d$ be the input features of the i-th layer of a CNN. The dot-product self-attention utilizes three different linear layers to transform ${\mathbf{f}}^i$ into three feature maps: query ${\mathbf{q}}^i\in {\mathbb{R}}^{d_k}$, key ${\mathbf{k}}^i\in {\mathbb{R}}^{d_k}$, and value ${\mathbf{v}}^i\in {\mathbb{R}}^{d_v}$. For matrix multiplication, the queries and keys must have the same feature dimension $d_k$. The similarity between the i-th query and j-th key was measured by $\rho \left({\mathbf{q}}^i{{\mathbf{k}}^j}^{\mathsf{T}}\right)$, where $\rho$ denotes a normalization function. Hence, the dot-product self-attention computes the similarities between all pairs of the position in the feature maps. By utilizing the relationships as weights, the output feature was obtained by the weighted summation of the values from all positions that were aggregated at the position i.

Assuming that all n positions’ queries, keys, and values in the matrix formed as $\mathbf{Q}\in {\mathbb{R}}^{n\times d_k}$, $\mathbf{K}\in {\mathbb{R}}^{n\times d_k}$, and $\mathbf{V}\in {\mathbb{R}}^{n\times d_v}$, the output of the dot-product self-attention is defined as

$$\mathbf{D}(\mathbf{Q},\mathbf{K},\mathbf{V})=\rho (\mathbf{Q}\otimes {\mathbf{K}}^{\mathsf{T}})\otimes \mathbf{V},$$

(11)

where ⊗ represents the matrix multiplication. There are two main choices for the normalization function $\rho$.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Scaling:& \rho \left(\mathbf{Z}\right)=\frac{\mathbf{Z}}{n}\hfill \\ \hfill Softmax:& \rho \left(\mathbf{Z}\right)={\delta }_{row}\left(\mathbf{Z}\right),\hfill \end{array}\end{array}$$

(12)

where ${\delta }_{row}$ is the adopted Softmax function in each row of matrix $\mathbf{Z}$.

The main obstacle to using this mechanism is the resource demands. This operation calculated the relationship between each pair of positions, which induced $n^2$ similarities. Hence, it yielded a memory and computational complexity of $O\left(n^2\right)$ and $O\left(d_k{n}^2\right)$, respectively, where O denotes the large O notation. Owing to the resource demands, this mechanism is mainly applied to low-resolution features.

Shen et al. [31] proposed an efficient self-attention mechanism that is mathematically identical to the dot-product self-attention but significantly faster and more memory-efficient. This mechanism also applies three linear layers to the input feature $\mathbf{F}\in {\mathbb{R}}^{n\times d}$ to form the queries $\mathbf{Q}\in {\mathbb{R}}^{n\times d_k}$, keys $\mathbf{K}\in {\mathbb{R}}^{n\times d_k}$, and values $\mathbf{V}\in {\mathbb{R}}^{n\times d_v}$. Unlike the dot-product self-attention mechanism, which solves the keys as n feature vectors in ${\mathbb{R}}^{d_k}$, the efficient module considers them as $d_k$ single-channel feature maps. This module then generates a global context vector by utilizing each of these feature maps as weights over all areas, and accumulating the value features through weighted summation.

Thus, the efficient self-attention mechanism is expressed as

$$\mathbf{E}(\mathbf{Q},\mathbf{K},\mathbf{V})={\rho }_q\left(\mathbf{Q}\right)\otimes ({\rho }_k{\left(\mathbf{K}\right)}^{\mathsf{T}}\otimes \mathbf{V}),$$

(13)

where ${\rho }_q$ and ${\rho }_k$ denote the normalization functions for the query and key features, respectively. The two normalization approaches are formulated as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Scaling:& {\rho }_q\left(\mathbf{Z}\right)={\rho }_k\left(\mathbf{Z}\right)=\frac{\mathbf{Z}}{\sqrt{n}}\hfill \\ \hfill Softmax:& {\rho }_q={\delta }_{row}\left(\mathbf{Z}\right),\hfill \\ \hfill & {\rho }_k={\delta }_{col}\left(\mathbf{Z}\right),\hfill \end{array}\end{array}$$

(14)

where ${\delta }_q$ and ${\delta }_k$ are the Softmax functions for each row or column of the matrix $\mathbf{Z}$, respectively.

The efficient self-attention is equal to the dot-product self-attention when the scaling normalization method is employed. Substituting the scaling normalization formula in Equation 12 into Equation 11 yields

$$\mathbf{D}(\mathbf{Q},\mathbf{K},\mathbf{V})=\frac{\mathbf{Q}\otimes {\mathbf{K}}^{\mathsf{T}}}{n}\otimes \mathbf{V}.$$

(15)

Similarly, substituting the scaling normalization formula in Equation 14 into Equation 13 produces

$$\mathbf{E}(\mathbf{Q},\mathbf{K},\mathbf{V})=\frac{\mathbf{Q}}{\sqrt{n}}\otimes (\frac{{\mathbf{K}}^{\mathsf{T}}}{\sqrt{n}}\otimes \mathbf{V}).$$

(16)

Because n is scalar and matrix multiplication is associative, Equation 16 is formulated as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathbf{E}(\mathbf{Q},\mathbf{K},\mathbf{V})& =\frac{\mathbf{Q}}{\sqrt{n}}\otimes (\frac{{\mathbf{K}}^{\mathsf{T}}}{\sqrt{n}}\otimes \mathbf{V})\hfill \\ \hfill & =\frac{1}{n}\mathbf{Q}\otimes ({\mathbf{K}}^{\mathsf{T}}\otimes \mathbf{V})\hfill \\ \hfill & =\frac{1}{n}(\mathbf{Q}\otimes {\mathbf{K}}^{\mathsf{T}})\otimes \mathbf{V}\hfill \\ \hfill & =\frac{\mathbf{Q}\otimes {\mathbf{K}}^{\mathsf{T}}}{n}\otimes \mathbf{V}.\hfill \end{array}\end{array}$$

(17)

As result, correlating Equation 15 and 17 yields $\mathbf{E}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathbf{D}(\mathbf{Q},\mathbf{K},\mathbf{V})$. A structural overview of the dot product and its efficient self-attention is shown in Figure 4.

Owing to the effective implementation and efficiency of the attention mechanism, we employed ESA at the decoder to extract long-range information on aggregated features and obtain clues from other areas to successfully fill-in the missing areas. The impact of ESA is examined in detail in Section 4.7.

3.2. Discriminator Architecture

For adversarial learning, we adopted a multiscale discriminator framework. The discriminator has a PatchGAN [52] structure with five $4\times 4$ vanilla convolution layers and step size of $\{2,2,2,1,1\}$. For the first three convolutional layers, the spatial size of each output feature map was halved, while the number of channels was doubled. The fourth convolutional layer also doubled the number of channels, while the last convolutional layer transformed the final output feature map into a map with one channel. For all convolutional layers, spectral normalization [50] was adopted to satisfy the 1-Lipschitz constraint and ensure the training process. The LReLU was used as the activation function with a hyperparameter of 0.2.

To apply a conditional setting for the learning process, the generated and target images were concatenated channel-wise with the corresponding binary mask $\mathbf{M}$, and fed into the discriminator as the input. In addition, both inputs were downsampled by half using nearest-neighbor interpolation and fed into another discriminator to satisfy the multiscale discriminator framework. The output sizes of the two discriminators are $30\times 30\times 1$ and $14\times 14\times 1$. A structural overview of the multiscale discriminator framework is shown in Figure 1.

3.3. Loss Function

To train our model, we used a combination of loss functions, namely, the adversarial loss [19], feature-matching (FM) loss [53], perceptual loss [54], style loss [55], and reconstruction loss.

First, for the adversarial loss, we employed the hinge loss [56] as the objective function for the generator, which is defined as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill L_{adv-G}^1=-{\mathbb{E}}_{[\mathbf{M},I_{out}]}& \left[D^1\left([\mathbf{M},I_{out}]\right)\right]\hfill \\ \hfill L_{adv-G}^2=-{\mathbb{E}}_{{[\mathbf{M},I_{out}]}_{\downarrow }}& \left[D^2\left({[\mathbf{M},I_{out}]}_{\downarrow }\right)\right],\hfill \end{array}\end{array}$$

(18)

where $I_{out}$ denotes the output image from the generator, ↓ is the nearest-neighbor interpolation downsampled by half, and $D^1$ and $D^2$ are discriminators with inputs of different scales. The objective functions of the two discriminators are expressed as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill L_{adv-D}^1={\mathbb{E}}_{[\mathbf{M},I_{target}]}\left[ReLU(1-D^1\left([\mathbf{M},I_{target}]\right))\right]+{\mathbb{E}}_{[\mathbf{M},I_{out}]}& \left[ReLU(1+D^1\left([\mathbf{M},I_{out}]\right))\right]\hfill \\ \hfill L_{adv-D}^2={\mathbb{E}}_{{[\mathbf{M},I_{target}]}_{\downarrow }}\left[ReLU(1-D^2\left({[\mathbf{M},I_{target}]}_{\downarrow }\right))\right]+{\mathbb{E}}_{{[\mathbf{M},I_{out}]}_{\downarrow }}& \left[ReLU(1+D^2\left({[\mathbf{M},I_{out}]}_{\downarrow }\right))\right],\hfill \end{array}\end{array}$$

(19)

where $I_{target}$ is the target image and $ReLU$ is the rectified linear unit (ReLU) activation function.

Second, we utilized the FM loss. The FM loss caused the generator to create more reasonable and realistic outputs by measuring the features of the output image $I_{out}$ and target image $I_{target}$ in each of the discriminators. Let $S^1$ and $S^2$ be the number of convolutional layers of $D^1$ and $D^2$; $E_k^1$ and $E_k^2$ the number of elements in the kth activation layer of $D^1$ and $D^2$; and $D_k^1$ and $D_k^2$ the activation diagram of layer k of $D^1$ and $D^2$, respectively. The FM loss is defined as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & L_{fm}^1=\mathbb{E}\left[\sum _{k=1}^{S^1}\frac{1}{E_k^1}\parallel D_k^1\left(I_{target}\right)-D_k^1\left(I_{out}\right){\parallel }_1^1\right]\hfill \\ \hfill & L_{fm}^2=\mathbb{E}\left[\sum _{k=1}^{S^2}\frac{1}{E_k^2}\parallel D_k^2\left(I_{target}\right)-D_k^2\left(I_{out}\right){\parallel }_1^1\right],\hfill \end{array}\end{array}$$

(20)

The mean absolute error (MAE) was adopted to compute the distance between features.

Third, we used the perceptual loss, which compares the feature maps acquired using the same convolution operation for the target and output images. This loss enables the generator to enhance the high-level semantic correlation between the two images by calculating and minimizing their differences. Specifically, we compared the distances between the activation features of the five layers (relu1-1, relu2-1, relu3-1, relu4-1, and relu5-1) of the target and generated images in the VGG-19 network [57] trained on the ImageNet dataset [58]. Thus, the perceptual loss is formulated as

$$L_{perc}=\mathbb{E}\left[\sum _k\frac{1}{E_k}\parallel {\tau }_k\left(I_{target}\right)-{\tau }_k\left(I_{out}\right){\parallel }_1^1\right],$$

(21)

where $E_k$ denotes the number of elements in the kth activation layer and ${\tau }_k$ denotes the activation diagram of the correlated layer.

Fourth, we applied the style loss, which is the correlation coefficient activation value of each activation feature channel. In our method, the VGG-19 network was used to extract feature maps as in $L_{perc}$. Its correlation is defined by computing the eccentric covariance between the diverse activation characteristic graphs of various scales. The style loss is expressed as

$$L_{style}={\mathbb{E}}_k\left[\parallel G_k^{\tau }\left(I_{target}\right)-G_k^{\tau }\left(I_{out}\right){\parallel }_1^1\right],$$

(22)

where k denotes the kth activation layer and $G_k^{\tau }$ is the Gram matrix ${\tau }_k$ with a size of $C_k\times C_k$.

Finally, we adopted the reconstruction loss to compute the distance between the corresponding pixels of the target and output images. As in the FM, perceptual, and style losses, the MAE was used to calculate the difference because it mitigated the problem of exploding gradients by maintaining a stable gradient for any input. The reconstruction loss is defined as

$$L_{rec}=\parallel I_{target}\odot (1-\mathbf{M})-I_{out}\odot (1-\mathbf{M}){\parallel }_1^1,$$

(23)

where only the restored areas were used to compute the loss.

Using all of the abovementioned losses, the overall loss function of our model is formulated as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill L_{PEIPNet}=& {\lambda }_{adv}(L_{adv-G}^1+L_{adv-G}^2)+{\lambda }_{fm}(L_{fm}^1+L_{fm}^2)+\hfill \\ \hfill & {\lambda }_{perc}{L}_{perc}+{\lambda }_{style}{L}_{style}+{\lambda }_{rec}{L}_{rec},\hfill \end{array}\end{array}$$

(24)

where the loss weights are set as follows: ${\lambda }_{adv}=1$, ${\lambda }_{fm}=1\times {10}^2$, ${\lambda }_{perc}=1\times {10}^2$, ${\lambda }_{style}=1\times {10}^2$, and ${\lambda }_{rec}=1\times {10}^3$.

4. Experiments

4.1. Datasets

The PEIPNet was trained on three public datasets: Paris StreetView [59], CelebA [60], and Places2 [61]. The Paris StreetView dataset contains 15,900 training samples and 100 testing images. The CelebA human face dataset contains approximately 160,000 training images and 19,900 testing images. The standard training dataset Places2 contains over four million images. Our model was evaluated on this validation dataset with 36,500 images. For training and testing, we followed the same procedure as in [14], in which random augmentation methods, such as random translation, rotation, dilation, and cropping, were used to augment the training masks, A mask set with 12,000 irregular masks that were pre-organized into six intervals according to the mask area ($1\%\sim 10\%,10\%\sim 20\%,\cdots ,50\%\sim 60\%$) was employed for testing. All the images in the Paris StreetView, CelebA, and Place2 datasets were resized to $256\times 256$ using bicubic interpolation.

4.2. Compared Methods

We compared our method with state-of-the-art models, such as EC [24], RFR [62], CR-Fill [63], CTSDG [64], and SPL [65]. All these models were pretrained; hence, their performance was directly evaluated in our settings.

4.3. Implementation Details

In the experiments, we used the Adam optimizer [66] (${\beta }_1=0.9$, ${\beta }_2=0.999$). The learning rates were initialized to $1\times {10}^{-3}$ and $4\times {10}^{-3}$ for the generator and discriminators, respectively. The batch size was set to 32. Our model was trained for $1\times {10}^5$ iterations, with the model evaluated at each $1\times {10}^3$ iteration. The cosine annealing algorithm [67] was selected as the learning rate scheduler to slowly decay the learning rates of the generator and discriminators to $1\times {10}^{-5}$ and $4\times {10}^{-5}$, respectively. The PyTorch framework [68] and an NVIDIA A100 GPU with 80 GB of RAM were utilized to implement the proposed method. (The official code will be uploaded on GitHub when this paper is accepted.)

4.4. Analysis of Model Complexity

Because we introduced a lightweight architecture for the image-inpainting task, the number of parameters and operational memory of each model was computed and compared, as listed in Table 1. Models with a batch size of one were used to measure the memory usage. For the model capacity, our method had less than one million parameters, which is the smallest among the models. The parameter difference ratio compared with other approaches ranged from 4.5 to 57.9. For the operational memory, PEPSI [21] would be an exceptional comparison, but was excluded because the official code was not provided by its authors. Our model also consumed the least amount of computational resources, i.e., approximately seven times lower compared with CR-Fill, the model with the second-lowest consumption. Having the most efficient structure in terms of both the number of parameters and operational memory provides a significant advantage in solving the overfitting problem, which commonly occurs when using a dataset with a limited number of images.

4.5. Qualitative Evaluations

Figure 5 shows the qualitative comparison results on the Paris StreetView dataset. The EC generated an edge map corresponding to the final output and identified the global arrangement and long-range features by employing dilated convolutional layers. However, minute textural details were not extracted, resulting in an offset in the local target. For example, in the first row, the straight line above the front sign was not recovered, which corrupted the distinct boundary. In the second to fourth rows, the texture of the tree, the structure of the bricks between the middle highest window, and the pattern of the building surface were distorted. RFR sequentially filled-in the missing pixels by circular feature inference, which generated high-fidelity visual effects; however, serious artifacts developed. For example, in the first to third rows, wrinkled patterns were produced in the large masked regions. In the fourth row, checkerboard artifacts were found on the bricks, which is a common problem of transposed convolutional layers [69]. CTSDG bound the texture and structure to each other; however, the boundaries were blurred owing to the implicit usage of the structure. For instance, in the first to third rows, the straight lines were obscured, which distorted the distinct boundaries of semantically different regions. In the fourth row, cross-patterned deformities developed throughout the region. SPL conducted knowledge distillation on the pretext models and modified the features for image-inpainting. These not only helped in understanding the global context but also provided structural supervision for the restoration of the local texture. Nonetheless, the local texture was still smoothed out, which resulted in blurring effects. For example, although solid lines were retained in all the rows, the texture of the leaves and brick patterns were not retained in the second to fourth rows, respectively. In contrast, our proposed method restored images with a suitable balance of low- and high-level features. For all rows, the pixels were filled with clear boundaries and a semantically plausible texture, as seen in the second row. This result is attributed to the use of ESA, where the model obtained hints on the texture from all areas of the corresponding feature maps.

Figure 6 shows the qualitative comparison results on the CelebA dataset. EC obtained unsatisfactory results, i.e., the facial structures were extremely distorted. For instance, in the first row, the position of the left eye was not symmetrical to the right eye. In the second row, the nose did not have the appropriate shape, while the mouth was barely visible. In the fourth row, although the eyes and nose had the proper silhouettes, the mouth was hardly seen. RFR provided better results than EC, but the final outputs did not improve. Although the model generated eyes with a normal shape, the mouths in all the rows were distorted, which ruined the degree of image restoration. CTSDG had the least favorable results. For all rows, the facial structures were not retained due to blurring effects, and checkerboard artifacts were found in all inpainted regions. SPL sufficiently recovered the images, but there were still some implausible regions remained. For instance, in the first row, the size of the left eye was different and relatively smaller than the right eye. In the fourth row, the wrinkles and beard on the face disappeared owing to excessive smoothing. In contrast, our model generated images with the best quality. For example, in the first row, the size of the left eye was similar to that of the right eye, and was at a suitable location. In the third row, unlike the other models, our method generated a mouth with teeth that was very close to the ground-truth image. In the fourth row, the wrinkles and beard with a proper mouth were retained, which has the least perceived difference compared with the other methods.

Figure 7 shows the qualitative comparison results on the Places2 dataset. EC restored images with an acceptable quality using an edge map; however, some areas were not appealing. For example, in the third row, the rails of the roller coaster were connected by curved lines, which is unrealistic. In the fourth row, the leaves filled with generated pixels did not have a consistent color compared with the other regions. CTSDG produced images with indistinct boundaries, i.e., unrealistic results. For instance, in the second row, the structure of the window was not fully retained owing to the blurriness of the bottom-left region. In the third row, the ride paths appeared disconnected, which is unrealistic. In the fourth row, the texture of the leaves contrasted with the other regions and was not harmonized with different areas. CR-Fill trained the generator by adopting an auxiliary contextual reconstruction task that made the generated output more plausible, even when restored by the surrounding regions. Hence, CR-Fill reconstructed images with an acceptable quality; however, some regions were still perceived as different. For instance, in the first and third rows, the boundaries of the trees were not obvious, and the color of the middle-right part of the ride was inconsistent. SPL produced outputs with distinct lines connecting the masked regions; however, key textures and patterns were lost owing to excessive smoothing. For example, in the first, third, and fourth rows, the textures of the objects were blurred. The generated image in the second row contained checkerboard artifacts that distorted the texture and quality of the image. In contrast to other methods, our proposed model achieved a balance between the apparent boundaries and textures of various objects. For instance, all the rows had straight lines that separated semantically different areas. Furthermore, the textures of the objects were effectively restored, leading to plausible results.

In summary, our proposed method effectively balanced low- and high-level feature restoration. This proves the generalizability of the proposed method based on qualitative evaluations.

4.6. Quantitative Evaluations

To analyze the inpainting results of our proposed method and those of other models, we applied four different metrics: Fréchet inception distance (FID) [70], learned perceptual image patch similarity (LPIPS) [71], structural similarity (SSIM), and peak signal-to-noise ratio (PSNR). The FID is a widely used quantitative metric in the field of image generation that measures the Wasserstein-2 distance between the generated and target images utilizing a pretrained Inception-V3 model [46]. Except for the FID, the other metrics are full-reference image quality assessments, in which restored images are compared with their corresponding ground-truth images. The LPIPS evaluates the restoration effect by computing the similarity between the deep features of two images using AlexNet [72]. The SSIM calculates the difference between two images in terms of their brightness, contrast, and structure. Finally, the PSNR analyzes the restoration performance by measuring the distances between the pixels of two images. The quantitative comparison results on the Paris StreetView, CelebA, and Places2 datasets are listed in Table 2, Table 3, and Table 4, respectively. For all the results, the first and second highest values are labeled in bold and underlined (↓ lower is better; ↑ higher is better).

On the Paris StreetView dataset, our PEIPNet method was ranked as the first or second for all metrics. For mask rates of $(0.1,0.2]$ and $(0.2,0.3]$, PEIPNet achieved the best results, similar to the FID and LPIPS. However, for mask rates of $(0.3,0.4]$ and $(0.4,0.5]$, RFR had the best results, similar to the FID and LPIPS, while PEIPNet had the second-best results. For SSIM and PSNR, SPL and PEIPNet had the best and second-best results, respectively, for all mask rates. The excellent performance of PEIPNet was attributed to the few artifacts in the generated images. The textures of different objects were also retained, which FID and LPIPS are highly sensitive to. Hence, PEIPNet can fill-in small masked regions, but its strength decreased in the large-hole inpainting task. This is because DDCM and ESA encouraged PEIPNet to obtain various meaningful hints from different regions of the feature maps with small masked areas by identifying global long-range and local features with dilated convolution and nonlocal attention. If there are insufficient regions from which to obtain information, the aforementioned mechanism reduced the performance of PEIPNet.

On the CelebA dataset, PEIPNet was ranked as the first or second best for the LPIPS, SSIM, and PSNR. EC had the best outcomes for the FID with all mask rates, followed by SPL. However, the FID difference between SPL and PEIPNet was very small, except with the mask rate of $(0.4,0.5]$. For the LPIPS, PEIPNet had the best results with the first three mask rates and the second-best with the highest mask rate. The opposite is true for the first and second-best results of the RFR. For the SSIM and PSNR, SPL had the best values, followed by PEIPNet. As on the Paris StreetView dataset, the disparity in inpainting performance compared with the best method continued to increase as the mask rate increased owing to the aforementioned reason.

On the Places2 dataset, PEIPNet again had either the best or second-best LPIPS, SSIM, and PSNR. Unlike on the CelebA dataset, PEIPNet also had the second-best outcome for the FID with mask rates of $(0.1,0.2]$ and $(0.2,0.3]$. For the LPIPS, PEIPNet had the lowest values with the first three mask rates, and the second lowest with the highest mask rate; the opposite is true for SPL. For the SSIM and PSNR, PEIPNet had the second highest values for all mask rates, while SPL had the best outcomes. The same phenomenon of the increased inpainting accuracy difference compared with the best result was also observed on the Places2 dataset.

The proposed PEIPNet method showed exceptional performance for all metrics: FID, LPIPS, SSIM, and PSNR. In most cases, PEIPNet had the best or second-best outcome; this tendency was not observed in the other methods. Specifically, PEIPNet achieved at least the second-best results on the Paris StreetView dataset, indicating the advantage of having a small number of model parameters when training with a limited number of samples. Thus, the quantitative evaluations confirmed the generalizability of the proposed method.

4.7. Ablation Studies

To verify the effects of the introduced the DDCM and ESA, ablation studies using our method were conducted on the Paris StreetView dataset. Specifically, we divided the DDCM into two parts for analysis: dilated convolution and dense block. To reduce the training time, we altered the batch size to eight for all combinations.

The quantitative results with different combinations of DDCM and ESA on the Paris StreetView dataset are listed in Table 5. For the DDCM, eliminating the entire module affected the model performance, where the average FID and LPIPS increased by 5.3607 and 0.0102, while the SSIM and PSNR decreased by 0.0083 and 0.4658, respectively, compared with the original model. Comparison of the two parts of the DDCM showed that applying dilated convolutional layers yielded better results for all metrics, which indicates the importance of long-range feature extraction in the image-inpainting task. ESA plays a crucial role because the average FID and LPIPS increased by 2.32 and 0.0034, while the SSIM and PSNR decreased by 0.0025 and 0.0676, respectively. However, the decline in the ESA performance was lower than that of the DDCM, indicating its dominance in the proposed method.

The qualitative results with different combinations of the DDCM and ESA on the Paris StreetView dataset are shown in Figure 8. Unlike the original model, the remaining combinations did not retain the streetlight structure. Specifically, the pillar was disconnected from the head of the lamp, which is unrealistic. The authentic model provided the best restoration of the texture of the leaves, demonstrating the strength of the proposed modules.

Finally, we calculated the complexities of different combinations of models, as described in Section 4.4 and summarized in Table 6. The contribution of dilated convolution was minor because there was almost no change in the memory when this process was eliminated. Removing the dense block had a greater impact on the memory compared with dilated convolution, but the change remained insignificant. On the other hand, eliminating ESA had a significant impact on the memory through a 4.51% reduction in the computational cost. Thus, adopting self-attention remains costly despite its structural efficiency.

5. Conclusion

In this study, we introduced the PEIPNet model to fill-in the missing pixels in damaged images. The proposed method has a one-stage inpainting framework, in which depthwise and pointwise convolutions were utilized to minimize the number of parameters and computational costs. We introduced three distinct modules into our model to produce semantically plausible outputs. First, SPADE was adopted to normalize the feature maps according to the input mask. Second, DDCM was used at each scale to generate pixels and capture global information along the activations. Finally, we employed ESA to extract long-range information and obtain references from undamaged areas. Our model consumed the lowest amount of operational memory compared with state-of-the-art models, demonstrating its potential. In addition, the experiments showed that the proposed method can be generalized in terms of both qualitative and quantitative perspectives. In future studies, we plan to refine PEIPNet to obtain higher-resolution images for real-world applications.