Diffusion Denoising Process with Gated U-Net for High-Quality Document Binarization

1. Introduction

Document images are essential data for digital document analysis such as OCR, historical document restoration and document classification [1]. However, document images obtained in real world are degraded document images due to various noise such as shadow, stain, ink smear, bleed-through, overwriting and variable background intensity [2]. These degraded document images have negative effect on various digital document analysis. Therefore, in order to effectively perform a digital document analysis, preprocessing of the degraded document image is essential. Document binarization is an important preprocessing task to obtain a clean binarized image by restoring a text region from a degraded document [3] . Document binarization does not simply remove specific noise from a degraded document image, but comprehensively processes various degradation factors [4]. However, converting a degraded document containing non-uniform noise into a clean binarized document is difficult to solve using traditional binarization algorithms as shown in Figure 1. Also, during this process, the difficulty of preserving elaborate text strokes is included. Due to these difficulties, research on document binarization of degraded document images is essential.

Research on document binarization has been ongoing for decades. Through decades of research, various image binarization algorithms [5,6,7] have been proposed, and they contributed to document binarization on normal and uniform images. However, it has limitations in distinguishing non-uniform and complex degradation and extracting elaborate text region in degraded document images [8]. To overcome these problems, deep learning-based computer vision methods have been applied to document binarization recently. Deep learning-based methods [9,10,11] such as segmentation and deep neural networks have been utilized. In [12], they proposed a model that redefined document binarization as a pixel classification problem by applying FCN (Fully Convolutional Network). In addition, in [8], the performance of document binarization was improved by training iterative neural network by combining the existing binary algorithm and deep learning. Recently, document binarization based on generative adversarial networks (GANs) has also been proposed. In [13,14,15], they successfully applied GAN as an image-to-image task to generate clean binarized document images from degraded document images. These deep learning-based methods overcame the limitations of traditional binarization algorithms and improved performance on text region preservation. However, these deep learning-based methods still have trouble extracting elaborate text strokes from non-uniform and complex degradation, and mode collapse [16], which degrades performance due to concentration on a specific data distribution, occurs.

To solve the above problems, we propose a document binarization model based on latent diffusion model [17], defining the document binarization as the iterative diffusion-denoising process as shown in Figure 2. Intuitively, this process generates a clean binarized document image, removing gaussian noise as each time step passes. To this end, we use the diffusion model [18], which has shown success in image generation tasks. Diffusion model is a model that performs image generation tasks by adding and removing gaussian noise in an image, and has the advantage of not relying heavily on training data by training to remove noise using both the distribution of data and time steps. Latent diffusion models showed improvement in time efficiency and precision of image generation by using the diffusion model in the latent space [17]. By applying this latent diffusion model to document binarization, the generalization effect on unseen noise is increased and the feature of text strokes is finely adjusted using the latent space. To the best of our knowledge, this is the first work introducing the diffusion model into document binarization. Additionally, a gated convolution is applied to the model backbone network for elaborate text stroke extraction. Gated convolution demonstrated its performance in binary mask learning on the segmentation models [19]. In this work, gated convolution is effective in distinguishing between text and background region by training feature through original convolution and gating value, which means text region information. We utilize gated convolution in latent space, not in pixel space, so that the proposed model extracts more elaborate text strokes.

To demonstrate the effectiveness of the proposed model, we conducted experiments on H-DIBCO, DIBCO (Document Image Binarization Contest) dataset [20,21,22,23]. For evaluation, we used four most commonly used metrics in document binarization and the proposed model outperformed the existing state-of-the-art methods in all metrics. In summary, this paper makes three significant contributions:

• We propose a novel approach by redefining document binarization as an iterative diffusion-denoising problem for degraded document images. This is the first work to introduce a latent diffusion model to document binarization to generate high-quality clean binarized document images.
• We use gated convolution in the latent space for elaborate text strokes extraction. This makes it easier to distinguish text from background by updating the gating value as a guide for the text region.
• The proposed model generates high-quality clean binarized document images and outperforms existing methods on several (H-)DIBCO datasets.

The rest of this paper is formulated as follows. In Section 2, related works are mentioned. Section 3 introduces the proposed model. Experimental results are discussed in Section 4. Conclusion are mentioned in Section 5.

2. Related Work

2.1. Document Binarization

Document binarization aims to perform pixel-wise binary classification of the background and text in a document image. Methods for document binarization can be divided into traditional binarization algorithm-based methods and deep learning-based methods. Traditional binarization algorithms use various nonparametric-based threshold algorithm. These methods perform binary classification through a threshold for background and text. Otsu’s method [6] uses a global threshold method that maximizes the distance between background and text and finds the maximum interclass variation for binary classification. These global threshold methods are effective for document images with uniform degradation, but are not suitable for document images with non-uniform degradation. To overcome this problem, pixel-wise local threshold methods have been proposed. Local threshold methods were used in [5,7], and more accurate classification was performed by measuring the relationship between nearby pixels based on local statistical information. These binarization algorithm-based methods are suitable at binarization of uniform noise, but still have difficulties in documents with various non-uniform noise such as overwriting [8].

To solve above problems, deep learning-based methods have been studied over the last few years. In [8], an iterative neural network was constructed to generate clean binarized images from non-uniformly degraded images, and Otsu’s algorithm was combined to the enhanced images. In [24], they succeeded in obtaining a binarized image by training mid-level representation through an encoder-decoder architecture consisting of CNN. Encoder-decoder architecture was also used to construct a network for selectional output in [25]. In [26], they proposed a cascading U-Net architecture for complex document image processing tasks. Akbari et al. [27] utilizes convolutional neural networks to identify foreground pixels using input-generated multichannel images. Also, generative models based on generative adversarial networks (GANs), which regard document binarization as an image-to-image task [28], have been proposed. Zhao et al. [13] proposed a clean binarized image generation model through multi-scale information combination based on conditional-GANs (cGANs). Also, [29] utilized cGANs and showed performance improvements in watermark removal, deblurring and binarzation. Lin et al. [30] proposed a 3 stage method for binarization by combining discrete wavelet transform and GAN. In [15], color-independent adversarial networks are constructed in two stages to learn global and local features of document.

2.2. Diffusion Model for image-to-image Task

Diffusion models [18,31,32] showed great success in image generation and are used in various image generation tasks. Rombach et al. [17] used the diffusion method in latent space to finely adjust semantic features of images to improve the quality of inpainting, super-resolution and image-to-image tasks. Also, diffusion models have been applied for image segmentation tasks. In [33], a distribution of segmentation masks was generated through stochastic sampling process. Kim et al. [34] proposed a diffusion adversarial representation learning model through switchable sptially-adative denormalization for vessel segmentation. Chen et al. [35] used the diffusion process on detection box proposals for object detection, and [36] used the diffusion process for image depth estimation.

2.3. Gated Convolutions

Gated convolutions are used for various tasks such as segmentation [37,38], inpainting [19], and language modeling [39]. Li et al. [37] proposed Gated Fully Fusion (GFF), which selectively fuses multi-level features using gated convolution in a fully connected way for segmentation. In [40], they proposed Context-Gated Convolution (CGC) that adaptively modifies the weights of convolutional layers according to the global context. Zhang et al. [41] used gated convolution for vessel segmentation. The network learns how to emphasize the edge of the vessel by utilizing gated convolution on the features extracted through the encoder-decoder architecture. In [19], a feature selection mechanism that can dynamically learn features for each spatial location is proposed to solve the problem of vanila convolution that uses the same filter for all input pixels. For the dynamic feature selection mechanism, gated convolution was used to effetively distinguish valid pixels from invalid pixels.

3. Method

We propose a binarization model of degraded document image through an iterative diffusion-denoising process. In this process, we use the latent diffusion model [17], which learns features of data distribution through the process of adding and removing gaussian noise to an image at each time step. The iterative diffusion-denoising process proceeds in the latent space through a pre-trained autoencoder and generates a clean binarized document image through the decoder of the autoencoder. In addition, by applying gated convolution in the denoising process, text stroke extraction performance is increased by updating the gating value corresponding to the text region.

In this section, first, the existing diffusion model is explained in the preliminaries, and then the architecture of the proposed model is explained. Next, the conditioned denoising process through the gated U-Net is described. Finally, the loss function of the proposed model is explained.

3.1. Preliminaries

Diffusion Models [18,31] are probabilistic models that aim to estimate a data distribution $p\left(x\right)$ by iteratively denoising noise from a normally distributed variable. The model performs various generative tasks through an iterative diffusion-denoising process. In the training stage, the diffusion model goes through a diffusion process that generates a noise vector $x_t$ by gradually adding gaussian noise $ϵ$ during the time step $t\in [0,T]$ from data $x_0$. The diffusion process $q(x_t\mid x_0)$ can be formulated as,

$$q(x_t\mid x_0):=\mathcal{N}(x_t\mid \sqrt{{\alpha }_t}{x}_0,(1-{\alpha }_t)I),$$

(1)

where ${\alpha }_t$ is a parameter for the variance schedule and is related to the degree of noise addition.

The denoising process generates a denoising vector $x_{t-1}$ from a random noise vector $x_t$ through a denoising network. Through this iterative process, $x_0$ is generated as a result. The denoising process $p_{\theta }(x_{t-1}\mid x_t)$ is as follows,

$$p_{\theta }(x_{t-1}\mid x_t):=\mathcal{N}(x_{t-1};{\mu }_{\theta }(x_t,t),{\sigma }_t^{2}I),$$

(2)

where ${\mu }_{\theta }(x_t,t)$ is a neural network to generate $x_0$ and learns iterative noise removal. ${\sigma }_t^2$ is a noise schedule and depends on $\alpha$.

The diffusion model trains the denoising network ${ϵ}_{\theta }(x_t,t)$ with equal weights, and the total loss $L_{DM}$ is formulated as follows,

$$L_{DM}={\mathbb{E}}_{x_0,ϵ,t}\left[\parallel ϵ-{ϵ}_{\theta }(x_t,t){\parallel }_2^2\right]$$

(3)

When sampling $x_0$, $x_0$ is generated using trained denoising network from the random noise vector $x_T$.

3.2. Network Architecture

Network architecture of the proposed model is based on the structure of latent diffusion models [17]. The overall architecture is shown in Figure 2. During the training process, a binarized document image given as input is converted into latent vector from the pixel space by the encoder of the pre-trained autoencoder. Utilization of the latent space makes it possible to finely adjust semantic features of latent vector [17,42,43]. Next, the diffusion process creates a gaussian noise vector by adding constant gaussian noise to the converted latent vector according to the time step. And, in the denoising process, the original latent vector is restored from the random noise vector through an iterative denoising network by injecting the damaged document image as condition. The completely denoised latent vector is converted into a cleaned binarized document image in pixel space through the decoder of the autoencoder. The proposed model is suitable for high-quality text region extraction as well as noise removal of damaged document images through an iterative diffusion-denoising process. It is demonstrated to achieve competitive performance through extensive experiments.

3.3. Document Diffusion-Denoising Network

3.3.1. Document Image Compression

The main process of the proposed model is performed in latent space, not in pixel space. To use the latent space, the document image in pixel space is converted into a latent vector through the autoencoder. In this process, we utilize $VQGAN$[43] using vector quantization layers. This method compresses a high-dimensional image vector through an encoder, and restores the compressed latent vector through a decoder absorbed by a vector quantization (VQ) layer to prevent information loss. In this process, it learns the classification of codebook that determines discrete latent vectors through the VQ layer. That is, by learning $\left|Z\right|$, the number of codebooks, the latent space is normalized by the VQ layer. Specifically, given an image of $x\in R^{H\times W\times C}$, the encoder E converts x into the latent vector $z\in R^{h\times w\times c}$, and the decoder D learns the process of restoring z back to $\tilde{x}$. $H,W,C$ mean height, width, and channel of the image vector, and $h,w,c$ mean height, width, and channel of the compressed latent vector, respectively. $VQGAN$ can preserve a specific region of an original image in the latent space by normalizing it to the latent space using the VQ layer. The latent space compression process of the proposed model compresses image data of 256× 256 size with a single channel into a latent vector of 64× 64 size with 3 channels. As a result, even if real image vector is converted to a latent vector via $VQGAN$, it is possible to preserve the text, background, and text boundary region like Figure 3.

3.3.2. Diffusion-Denoising Process

The iterative diffusion-denoising process of the proposed model is based on diffusion models [18,31] and proceeds according to equation 1 and equation 2 in latent space. The diffusion process of the network generates the noise vector $z_t$ by gradually adding the gaussian noise $ϵ$ to the latent vector $z=E\left(x\right)$ generated by the encoder. Then, in the denoising process, $z_0$ is restored by gradually removing noise from the noise vector $z_t$ through the gated U-Net architecture, which is called a denoising network. To this end, training proceeds by calculating the difference in distribution of the outputs of each process at the same time step. Gated U-Net architecture and its process are shown in Figure 4.

However, in the case of the unconditional diffusion-denoising process, it is trained with data consisting only of binarized document images. Accordingly, a document image following a distribution similar to that of the binarized document image can be generated, but a degraded document image cannot be restored to a cleaned binarized document image. For document binarization, the model should not only generate binarized document image, but also generate clean binarized document image under the condition of degraded document image. Therefore, as shown in Figure 2, we configure the conditional diffusion-denoising process by adding degraded document images as condition. The loss function for training the denoising network ${ϵ}_{\theta }$ with the condition added is as follows,

$$L_{LDM}={\mathbb{E}}_{E\left(x\right),c,ϵ,t}\left[\parallel ϵ-{ϵ}_{\theta }(z_t,t,E\left(c\right)){\parallel }_2^2\right]$$

(4)

To perform document binarization, in the denoising process, the latent vector $E\left(c\right)$ of the degraded document image c generated through the encoder is channel-wise concatenated as a condition. Also, we use skip connection to prevent information loss. When sampling a cleaned binarized document image, the latent vector of the degraded document image is concatenated with a random noise vector, and the noise is removed through a denoising network iteratively, trained as much as the time step.

3.4. Gated Convolution in Latent Space

We design the denoising network with gated U-Net architecture composed of fully gated convolution for elaborate extraction and seperation of text and background regions. Gated convolution proposed in [19] is a convolution that utilizes a dynamic feature selection mechanism that can learn features in each channel at each spatial location. Unlike the conventional convolution that multiplies the same filter on every spatial location, each location can be distinguished as a valid location and an invalid location by multiplying different spatial locations of the feature map with a weight called a gating value. The gating value is a value that allows more concentration on the text and text boundary region and has a value between 0 and 1 depending on each spatial location. Therefore, by passing the gating value of the feature map calculated in the previous layer to the next layer, it is possible to continuously give a guide on the valid location. In this work, valid location means text and text boundary region. In a specific region $(x,y)$ of a specific channel, the gated convolution can be formulated as,

$$Gatin{g}_{x,y}=W_g·I,$$

(5)

$$Featur{e}_{x,y}=W_f·I,$$

(6)

$$O_{x,y}=\varphi \left(Featur{e}_{x,y}\right)\odot \sigma \left(Gatin{g}_{x,y}\right),$$

(7)

where $W_g$ and $W_f$ are trainable convolution filters for extracting gating values and features, respectively, and I means an input feature map. $\sigma$ is the sigmoid function and $\varphi$ is the nonlinear activation functions. Therefore, the final output $O_{x,y}$ is calculated by element-wise multiplication of the value taking the sigmoid on the gating value and the value taking activation on the extracted feature value.

We use this gated convolution to construct a denoising network. In the denoising process, we design the network to extract elaborate text region by guiding information about the text and text boundary regions. The denoising network has a U-Net architecture and utilizes gated convolution in downsampling and upsampling processes. All downsampling consists of gated convolution, and upsampling performs upscaling through a scale factor and downsampling again. Therefore, the denoising network can be viewed as a fully gated convolution. Gated U-Net can effectively distinguish text region from background region in the latent space where each region is preserved by continuously injecting guides for each region through trainable gating values.

It is difficult to maximize the effect of gated U-Net, which consists of fully gated convolution, only with the loss of existing diffusion models that give training direction through the distribution of latent vector. Therefore, pixel-level loss $L_{pix}$ is added for text region extraction and effective updates of gating values. This calculates the difference between the result of the decoder at a specific time step and the ground-truth in pixel space. We add two notable pixel-level losses to update the difference between $\widehat{y}$, the output of the denoising network, and the ground-truth y, at a specific time step. The first is the binary cross entropy loss for binary value classification at the pixel level. The second is the dice loss, which computes the similarity between the predicted region of the model and the ground-truth region. Through this, $L_{pix}$ between the predicted pixel $\widehat{y}$ and the ground-truth pixel y is configured as follows, and $L_{pix}$ can be formulated as equation 10,

$$L_{bce}=-ylog\left(\widehat{y}\right)-(1-y)log(1-\widehat{y}),$$

(8)

$$L_{dice}=1-\frac{2y\widehat{y}}{y+\widehat{y}},$$

(9)

$$L_{pix}=L_{bce}+L_{dice}$$

(10)

The proposed model is trained to minimize $L_{total}$, weighted sum of $L_{LDM}$ and $L_{pix}$.

$$L_{total}=L_{LDM}+\lambda L_{pix}$$

(11)

The network is updated according to equation 11. However, since pixel-level loss can be less effective at lower time step, $\lambda$ is set as a penalty depending on the time step.

4. Experiments and Results

4.1. Dataset and Implementation details

We build a new training dataset through several existing document binarization datasets to evaluate and compare the performance of the proposed model with other methods. Training dataset consists of H-DIBCO [44,45,46], DIBCO [47,48,49], Bickley-diary dataset [50], Persian heritage image binarization dataset (PHIDB) [51], and Synchromedia Multispectral dataset (S-MS) [52]. Since most of the document images have a large size, we divided each image into 256 × 256 patches for train efficiency. Random rotation augmentation was performed on the divided patches to construct a dataset with a total of about 160k images. 90% of the built dataset was used as a training set, and 10% used as a validation set. Finally, for evaluation, H-DIBCO 2016 [20], DIBCO 2017 [21], H-DIBCO 2018 [22], DIBCO 2019-B [23] is used. The test dataset is not included in the training and validation set.

To train the proposed model, the time step of the diffusion-denoising process is set to 1000, and the AdamW optimizer [53] set to the initial learning rate $lr=1\times {10}^{-6}$ is used. In addition, the autoencoder was pre-trained for 20 epochs for the train set, and was frozen during denoising network training. We use $LeakyReLU$ as the network’s activation function. The denoising network was finally trained for about 1M steps with a batch size of 2. We use NVIDIA RTX 3090 GPU (24GB)×3 for training.

4.2. Evaluation Metrics

For quantitative evaluation of the proposed model, a total of four evaluation metrics [20,21,22,23] suitable for document binarization evaluation are selected. Metrics consist of F-measure (FM), pseudo-Fmeasure (pFM), Peak Signal-to-noise Ratio (PSNR), and Distance Reciprocal Distortion (DRD), commonly used in Document Image Binarization Contest (DIBCO).

F-measure is calculated using the precision and recall between the predicted pixel and the ground-truth pixel as,

$$FM=\frac{2\times Precision\times Recall}{Precision+Recall}$$

(12)

pseudo-Fmeasure was proposed in [54], and the stroke predicted through $pseudo$$Recall\left(pRecall\right)$ representing the percentage of the skeletonized ground-truth image and the stroke of the ground-truth are calculated as follows,

$$pFM=\frac{2\times Precision\times pRecall}{Precision+pRecall}$$

(13)

PSNR is an image quality evaluation metric and is calculated for the similarity between the predicted image and the ground-truth. PSNR is calculated as follows, where C is the maximum value of an image pixel,

$$PSNR=10log\frac{C^2}{MSE}$$

(14)

DRD is a metric proposed by [55] to measure visual distortion in binary document images. It measures the distortion for all the S flipped pixels as follows,

$$DRD=\frac{\sum _{k=1}^{S}DR{D}_k}{NUBN},$$

(15)

where $NUBN$ is the number of the non-uniform (not all black or white pixels) 8 × 8 blocks in the ground-truth image, and $DR{D}_k$ is the distortion of the k-th flipped pixel as defined in [55].

4.3. Quantitative and Qualitative Comparison

To demonstrate the performance and effectiveness of the proposed model, we use a total of four benchmark datasets. DIBCO 2016, DIBCO 2017, and DIBCO 2018 datasets are made up of machine-printed and handwritten document images, and DIBCO 2019-B is a challenging benchmark dataset that includes papyrus-like materials and extreme degradation. We use four evaluation metrics of FM, pFM, PSNR, and DRD introduced in Section. Section 4.2. In addition, for fair comparison, a total of 8 models including the proposed model are reimplemented. We used publicly available source codes provided by the authors for reimplementation. Methods compared with the proposed model can be divided into traditional binarization algorithm-based methods and deep learning-based methods. Binarization algorithm-based methods include Otsu [6], and Sauvola [5], and deep learning-based methods include SAE [25], cGANs [13], Akbari et al. [27], Souibgui et al. [29], and Suh et al. [15]. Since the proposed model is a generative model, we compare with various generative models. Additionally, the performance of competition winners of each year [20,21,22,23] is added to the experimental results.

Table 1 shows the quantitative evaluation results of models. Compared to other methods, the proposed model achieved the best performance for the mean values on all datasets. The proposed model shows best or second best performance in terms of pFM and PSNR on all four datasets. Especially in DIBCO 2019-B dataset, which consists of the most complex and noise that the model has not experienced, the proposed model achieves the best performance in all metrics. In particular, existing methods show great decrement, while the proposed model shows impressive performance gain compared to others even in such severe noise. Although the quantitative differences between the proposed model and the others are marginal on other datasets, qualitative evaluation shows the proposed model performs better at document binarization. Therefore, it demonstrates that training through the iterative diffusion-denoising process is effective in removing complex noise and robust to various environments.

First, the proposed model for H-DIBCO 2016 dataset shows the highest performance in pFM and the second highest performance in PSNR. In DIBCO 2017 dataset, the proposed model shows the highest performance in pFM and PSNR. The high performance of the proposed model in pFM means that it extracts the text stroke best compared to other methods, and high performance in PSNR means that it produces high-quality results. In H-DIBCO 2018 dataset, the proposed model achieve the second highest performance in all four metrics. Results on DIBCO 2019-B dataset show significantly higher performance in all four metrics compared to other methods. DIBCO 2019-B dataset is a challenging dataset consisting of images with extremely severe degradation on materials such as papyrus and tree bark. Even in this challenging dataset, the proposed model shows the highest performance in all metrics. However, other models show extremely low performance on this dataset compared to other datasets. This demonstrates that the proposed model successfully extracts text strokes even in extremely degraded document images that have not been trained. Average results for DIBCO 2016, 2017, 2018, and 2019-B datasets show that the proposed model achieved the highest performance in terms of all of the metrics. That is, we demonstrate the effectiveness of the proposed model for performing image generation through diffusion-denoising process and text stroke extraction through gated U-Net.

Qualitative evaluation on H-DIBCO 2016 dataset is shown in Figure 5. Shown image is the most challenging document image with complex degradation in H-DIBCO 2016 dataset. Figure 5g, the result image of [13], shows the highest quantitative result, but shows limitations in preserving elaborate text stroke. Figure 5i, the result image of [15], shows better results in preserving text strokes, but has limitations in removing noise. Figure 5j, the result image of the proposed model, shows that it not only shows high results in noise removal such as overwriting compared to other methods, but also performs best in preserving elaborate and accurate text strokes.

Figure 6 is the result image for DIBCO 2017 dataset for qualitative evaluation. The image contains noise that is difficult to distinguish between text and background. Traditional algorithm-based methods suffer from misclassifying such noise as text. In addition, deep learning-based methods also suffer from misclassification in the case of severe noise. It is difficult to distinguish this noise because it has a similar shape, size, and text stroke, but Figure 6j shows that the proposed model successfully distinguishes background and text region.

The result image for H-DIBCO 2018 dataset is shown in Figure 7. H-DIBCO 2018 dataset consists of 10 handwritten document images with various degradations such as background intensity, shadow, and bleed-through. Figure 7 includes degradation of variable background intensity and ink smearing. Other methods have difficulty in removing these degradations. In the case of Figure 7g, which is the result image of [13], degradation is relatively resolved well, but it has difficulty in preserving precise text strokes such as small footnotes in the document. In the case of the proposed model in Figure 7i, it solves all types of degradation well and shows high performance in preserving elaborate text strokes.

Figure 8 shows the result image for DIBCO 2019-B dataset. DIBCO 2019-B dataset consists of ancient document images that reflect various types of papyrus quality, ink and handwriting styles. In particular, it is challenging data that includes non-homogeneous properties such as resolution, lighting, and noise. Figure 8c and d, which are results of [13,15], cannot effectively distinguish text from background region according to the intensity of text and background. Because of this problem, it is extremely difficult to preserve text strokes. As shown in Figure 8e, the proposed model is effective for text and background classification and successfully preserves the text region. In addition, the types of degradation included in DIBCO 2019-B data are types that have not been experienced in the training data, and we demonstrate the proposed model to be effective even in severe degradation.

Additionally, Figure 9 shows that the proposed model is effective for elaborate region extraction. It is difficult to distinguish between background and text regions and extract accurate and elaborate text strokes from small parts like the example image. In this case, even in methods with relatively high quantitative evaluation [13,15,27] have difficulties in precisely extracting and preserving text regions, as shown in Figure 9c, d and e. However, in the case of the proposed model, as shown in Figure 9f, it successfully extracts and preserves precise text strokes despite these difficulties.

4.4. Ablation study

We perform ablation experiments to evaluate each component of the proposed model. For evaluation, DIBCO 2016 benchmark dataset is used and the effectiveness of the proposed model is demonstrated using the four metrics in Section 4.2. The baseline is [17], and the diffusion-denoising process is performed in the latent space using an autoencoder. We compare the baseline with each of the proposed gated U-Net and pixel-level loss, and models with all of the components. A total of four models are compared, and experiments are performed in the same implementation setting except for each component of each model.

Table 2 shows results of the baseline, the proposed model without $L_{pix}$, the proposed model without gated U-Net, and the proposed model. When gated U-Net is added to the baseline, pFM (95.03) improves by 0.59 compared to the baseline (94.44). Text stroke preservation performance has been improved because filter training for gating values is additionally included. However, when pixel-level loss is not included, the feedback for gated convolution filter is not effective, so there is no significant performance improvement. When gated convolution was excluded, text and background regions are updated through pixel-level loss, resulting in improved performance in all metrics. Finally, in the case of the proposed model with pixel-level loss for training the gated convolution filter in gated U-Net, FM, pFM, PSNR, and DRD improved by 0.68, 1.00, 0.28, and 0.31, respectively, compared to the baseline. This demonstrates that the components of the proposed model are effective respectively, and all components are effectively connected for performance gain.

Additionally, to confirm the effect of each component, the results of model training after six epochs are shown in Table 3. We adopt FM which verifies the predicted text and background pixel-wise, and pFM which verifies each prediction region. When pixel-level loss is excluded from the proposed model, each metric is similar to or slightly increased from the baseline. The proposed model without gated U-Net indicates the model with the baseline’s backbone network using LDM and pixel-level loss to update denoising network. This results in a significant improvement of FM by 8.00 and pFM by 7.94 compared to the baseline. In the case of the proposed model using gated U-Net for document binarization and pixel-level loss aimed at training gating values, it achieves the highest performance of each metric and shows a significant improvement with FM by 8.22 and pFM by 8.51 compared to the baseline. These results demonstrate that the components of the proposed model are effectively suitable for text stroke preservation and extraction.

Figure 10 shows the result images of the baseline, the proposed model without $L_{pix}$, the proposed model without gated U-Net, and the proposed model. We confirm that Figure 10f, a result of the proposed model with both $L_{pix}$ and gated U-Net, is successful in distinguishing text and background of degraded document images compared to Figure 10c, a result of baseline.

Additionally, in Figure 11, the elaborate text stroke preservation performance can be qualitatively confirmed. In Figure 11f, which is the result of the proposed model, noise removal and text stroke preservation are more accurate and precise than the baseline shown in Figure 11c. This proves that the diffusion-denoising process through gated U-Net and $L_{pix}$ is effective in preserving more elaborate text strokes and generates high-quality results.

5. Conclusion

In this paper, we propose a new document binarization model based on latent diffusion models that enables elaborate text stroke extraction and is robust against unexperienced degradation. The diffusion-denoising process in latent space is utilized for the first time to generate high-quality cleaned binarized document image. In addition, denoising network is constructed with gated U-Net consisting of a fully gated convolution for text stroke preservation of severe degraded document images. In this process, we add a filter to update the gating value to preserve text region. Also, by using pixel-level loss to effectively update the filter for the gating value, text strokes are effectively preserved even in extremely degraded images. We confirmed this strength through extensive experiments on challenging benchmark datasets. According to the quantitative experimental results, the proposed model shows impressive performance gain in text stroke preservation and extraction compared to existing methods in (H-)DIBCO dataset. Through various qualitative evaluations, we confirm that the proposed model overcomes the limitations of existing methods and successfully performs text and background classification. In future research, we expect that the proposed model can be used for various tasks that utilize binary masks.