A Hybrid Template-Guided Deep Learning Framework for OCR-Oriented Restoration of Degraded Tax Documents

1. Introduction

The restoration and binarization of degraded documents constitute critical problems in computer vision and image processing. However, for legal and tax documents, the requirements are particularly stringent: restoration must preserve content fidelity, as the loss or alteration of a single digit may invalidate a record and produce unacceptable errors in Optical Character Recognition (OCR). Consequently, purely visual restoration is insufficient, and both structural integrity and functional readability must be preserved with high precision.

In the Colombian context, the legal requirement to retain original supporting documents for extended periods increases the prevalence of physically deteriorated documents affected by mixed degradations. More generally, a significant proportion of historical and administrative documents exhibit degradation, limiting accessibility, analysis, and information preservation [1]. Unlike handwritten manuscripts or general-purpose administrative records, tax forms exhibit highly rigid layouts containing predefined fields (totals, Tax Identification Number (NIT), dates), tables, signatures, and encoded elements. As a result, any geometric distortion or local alteration may propagate and ultimately lead to systematic OCR failures.

Traditional methods, such as adaptive thresholding, have been widely adopted because of their simplicity and computational efficiency. However, their performance tends to deteriorate under complex noise, shadows, stains, and severe information loss [2]. Deep learning methods—particularly generative models—have demonstrated significant advances in document restoration. For example, EHDI employs Generative Adversarial Networks (GANs) to enhance the quality of degraded documents [1], while recent approaches such as DocDiff address deblurring, denoising, and artifact removal within a unified diffusion-based framework [3]. Similarly, Doc-Attentive-GAN incorporates attention mechanisms to guide the denoising process [4].

Nevertheless, these approaches present limitations when applied to structured tax documents. Generative models optimized primarily for perceptual quality may introduce spurious strokes or modify critical details such as characters, separators, and table boundaries, thereby affecting semantic consistency and OCR performance. Consequently, a gap remains between visually appealing restoration and restoration that is valid for automatic reading under strict layout constraints.

In this work, we propose a hybrid structure-guided restoration method for degraded tax documents. The approach begins with geometric normalization based on SIFT and RANSAC to map each form into a common canonical space defined by a master template. Within this aligned space, restoration is performed using a supervised model that incorporates explicit structural information through template priors, template fusion, and masked constraints focused on variable-content regions. By reducing restoration uncertainty in rigid-layout documents, the proposed framework aims to preserve form consistency and improve OCR utility without introducing severe computational overhead. To validate the proposed method, we compare it against both classical and learning-based baselines using visual, structural, textual, and practical efficiency metrics.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 describes the materials and methods, including synthetic dataset generation, the geometric normalization stage, and the proposed hybrid framework. Section 4 presents the experimental results, covering dataset validation, baseline comparisons, computational cost analysis, and ablation studies. Section 5 discusses the implications of the findings, the limitations of the proposed approach, and its applicability to sensitive document-processing scenarios. Finally, Section 6 summarizes the main conclusions of the study.

2. Related Work

2.1. Document Restoration

The restoration of degraded documents aims to improve readability and visual quality while preserving the integrity of textual content. This task becomes particularly challenging in the presence of mixed degradations, including noise, blur, stains, bleed-through artifacts, and non-uniform illumination [1]. In the recent literature, document restoration approaches can be broadly categorized into three main families: generative models (GANs and their variants), diffusion-based models, and analytical methods such as partial differential equations (PDEs) and filtering techniques, alongside complementary research directions including super-resolution and restoration guided by external information.

Adversarial models have been widely employed as image-to-image mapping frameworks to recover clean documents from degraded observations. Enhancement of Historical Document Images (EHDI) [1] is an early landmark in the enhancement of historical documents, including severely degraded samples, through the combination of multiple loss functions and adversarial training. In scenarios lacking aligned image pairs, CycleGAN (Cycle-Consistent Generative Adversarial Network) [5] has demonstrated competitive performance in denoising tasks while requiring lower training costs. Variants such as adversarial autoencoders [6] further improve efficiency by combining reconstruction and adversarial objectives, incorporating geometric and baseline normalization mechanisms to stabilize outputs. To focus restoration on relevant regions, Doc-Attentive-GAN introduces attention maps that guide the denoising process [4], while multi-scale approaches with style perception mechanisms, such as Style Perception—Multi-Scale Attention—Image Restoration (SP-MSA-IR) have reported improvements in clarity and resolution compared with traditional alternatives [7].

Diffusion models have gained increasing attention due to their ability to preserve high-frequency components and handle multiple degradations within a unified framework. DocDiff introduces a coarse predictor coupled with high-frequency residual refinement for deblurring, denoising, and watermark removal tasks [3]. In a hybrid direction, the use of cascaded Conditional Generative Adversarial Networks (cGAN), alternated with anisotropic diffusion enables the joint treatment of background and textual content, proving particularly effective in cases of severe bleed-through degradation [8].

PDE-based methods remain attractive because of their interpretability and their independence from large training datasets. They are particularly effective for illumination correction and contrast enhancement under physically motivated assumptions. Previous studies have proposed combinations of shock filtering and selective diffusion for addressing blur, noise, and ink artifacts [9] as well as illumination-specific models capable of correcting shadows, although such methods may face limitations around edges and specular highlights [10]. Other approaches combine illumination estimation with higher-order PDEs to restore both text intensity and edge information [11], or integrate spatiotemporal regularization and local entropy to improve text–background separability [12]. Additionally, p-Laplacian-based formulations have been explored to mitigate the effects of non-uniform illumination [13].

Regarding super-resolution and integrated restoration approaches, document enhancement also benefits from these strategies. Previous studies have reported the use of interpolation combined with wavelet transforms followed by GAN-based reconstruction to recover spatial and spectral details [14], as well as super-resolution networks incorporating Spatially-Adaptive Denormalization (SPADE) to improve edge quality and transition regions [15]. Finally, several works have integrated restoration with information extraction. These approaches combine denoising with models such as CycleGAN and Mask R-CNN to identify document fields and extract relevant textual information [16], or condition restoration on a target transcription to recover degraded content without paired training samples [17]. Along the same line of practical robustness, convolutional autoencoders incorporating residual blocks and skip connections have been proposed for denoising and deblurring tasks, achieving improved learning stability [18].

2.2. Document Binarization

Document binarization converts document images into foreground (text) and background representations. Its difficulty increases substantially in the presence of degradations affecting contrast, fine contours, and bleed-through artifacts, all of which directly impact subsequent OCR performance [19]. In general, the literature can be grouped into GAN-based approaches, diffusion/PDE-based methods, and alternative architectures designed to improve contour preservation and robustness.

Within GAN-based approaches, variants employing dual discriminators have been proposed to capture both global consistency and local detail [20,21]. Other research directions incorporate transform-domain representations (e.g., wavelets) to strengthen texture and color separation, as exemplified by CCDWT-GAN, which has achieved strong performance on DIBCO and H-DIBCO benchmarks [22], as well as two-stage frameworks integrating wavelet processing with adversarial learning [23]. Architectures incorporating Handwritten Text Recognition (HTR) modules within GAN frameworks have also been explored to restore degraded documents and improve readability in H-DIBCO challenges [24]. In terms of representation efficiency, quaternion-based layers combined with dual discriminators have been proposed to achieve compact architectures without compromising output quality [25]. Furthermore, two-stage adversarial models have demonstrated strong capabilities in foreground extraction while effectively capturing both global and local features [26]. Comparative studies have reported distinct performance characteristics among DE-GAN, DP-LinkNet, and 2-StageGAN on DIBCO benchmark datasets [27]. Finally, cascaded cGAN generators formulate binarization as an image generation problem, emphasizing foreground structural consistency [28].

Regarding diffusion and PDE-based binarization approaches, several PDE formulations and non-local operators have been proposed. These include non-local p-Laplacian models designed to preserve textual details under varying illumination conditions, albeit with sensitivity to preprocessing choices and parameter settings [19]. In the context of nonlinear diffusion, researchers have proposed equations incorporating selective source terms and binary projection mechanisms, achieving effective results on DIBCO benchmarks [29], as well as variants with adaptive source terms for enhanced edge preservation [29]. Other methods combine diffusion with filtering techniques, such as selective shock filters and time-dependent source functions [30] or integrate fuzzy logic into PDE formulations to address extreme staining conditions. More recently, cold diffusion frameworks have been explored to directly generate binarized outputs from degraded documents [31], while nonlinear diffusion models incorporating reaction terms have been implemented using finite-difference schemes and variable-splitting techniques [32]. In a related direction, binarization has been formulated as the outcome of an iterative denoising process based on latent diffusion and gated U-Net architectures [2]. Other relevant approaches include the combination of variational inference with convolutional neural networks to improve binarization robustness, including the generation of synthetic degradations during training [33]. Additionally, models such as Gated Convolutions-Based Document Binarization (GDB) have been proposed to improve precise foreground delineation through two-stage architectures with refinement mechanisms [34].

2.3. Research Gaps and Challenges

Although deep learning methods generally outperform classical approaches in complex scenarios, several key challenges remain: (i) generalization to out-of-distribution mixed degradations; (ii) the computational cost of diffusion-based methods compared with lighter alternatives; (iii) the gap between perceptual evaluation and functional evaluation; and (iv) the scarcity of datasets specifically designed for highly structured documents with strict layout constraints, such as tax forms.

In the context of tax documentation, we argue that a restoration framework capable of reducing the degrees of freedom of the restoration process through explicit structural and layout constraints is necessary. Such constraints can prevent undesired modifications to document content while preserving OCR utility in highly formal and structurally constrained documents.

3. Materials and Methods

The methodology is composed of two components that combine Digital Image Pro-cessing (DIP) and Deep Learning (DL). The first component is a synthetic data generation pipeline that produces supervised pairs (degraded document and its corresponding ground truth) under controlled degradations. The second component is a hybrid restora-tion method that explicitly separates geometric normalization from content restoration, first projecting the document into a canonical space and then applying structure-conditioned inpainting. Figure 1 summarizes the training, inference, and evaluation pipeline.

3.1. Synthetic Data Generation

The objective of the synthetic generator is to construct a supervised dataset with controlled degradations, enabling reproducible training and evaluation using a fixed seed and explicit per-sample parameter logging. To achieve this, the pipeline initializes from a base PDF template, from which a reference template image ($I_{template}$) and a master mask ($M_{master}$) are generated to isolate and focus on specific document regions. For each document $i$, the pipeline generates: (i) a clean/ground-truth/canonical image $I_{\mathrm{gt}}^{\left(i\right)}$, (ii) a degraded image $I_{\mathrm{noisy}}^{\left(i\right)}$ and (iii) a JSON file containing the full parameterization of the document (${\theta }_{\mathrm{fill}}^{\left(i\right)}$) and applied degradations (${\theta }_{\mathrm{deg}}^{\left(i\right)}$) (see Figure 2).

3.1.1. Form Rendering and Document Content

Starting from a PDF template of the selected form type, a filled PDF is generated using a document filling module that produces coherent field values (e.g., identifiers, dates, and amounts), returning a parameter dictionary ${\theta }_{\mathrm{fill}}^{\left(i\right)}$ for auditing and reproducibility. This PDF is rasterized into an image via PDF-to-image conversion, producing the canonical grayscale representation:

$$I_{\mathrm{gt}}^{\left(i\right)}=\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{y}\left(\mathrm{p}\mathrm{d}\mathrm{f}2\mathrm{i}\mathrm{m}\mathrm{g}\left(PD{F}_{\mathrm{filled}}^{\left(i\right)}\right)\right)$$

(1)

The canonical image is stored as a reference “without degradations”.

3.1.2. Degradation Operator: Transformation Chain

The degradation process is implemented as an ordered composition of operators, sampled from uniform or discrete distributions and activated according to configurable probabilities. Let $T_g$ denote geometric transformations (translation/rotation/scale), $T_{\mathrm{dpi}}$ denote resampling simulating DPI variation, and $T_n,T_{bc},T_b,T_{thr}$ denote noise, brightness/contrast adjustment, blur, and thresholding operators, respectively:

$$I_{noisy}^{\left(i\right)}=T_{thr}\left(T_b\left(T_{bc}\left(T_n\left(T_{dpi}\left(T_g\left(I_{\mathrm{gt}}^{\left(i\right)}\right)\right)\right)\right)\right)\right)$$

(2)

The image $I_{\mathrm{noisy}}$ includes the full degradation pipeline. The complete sampling and activation parameters for each transformation are detailed in Table 1.

3.1.3. Data Persistence and Sample Traceability

For each document $i$, the pipeline stores: $I_{\mathrm{gt}}^{\left(i\right)}$ (clean/GT/canonical), $I_{\mathrm{noisy}}^{\left(i\right)}$ (degraded), and a JSON file containing ${\theta }_{\mathrm{fill}}^{\left(i\right)}$ and ${\theta }_{\mathrm{deg}}^{\left(i\right)}$ (degradation parameter set). This ensures full reproducibility and auditability, as each sample can be regenerated from its JSON configuration and the seed.

3.1.4. Reproducibility and Parallelization

The generator supports a global seed $s$. In parallel execution, a per-document seed is defined as $s_i=s+i$, ensuring deterministic generation regardless of process scheduling order.

3.2. Hybrid Restoration Method

3.2.1. Canonical Space Formulation and Structural Guidance

The proposed method operates in a canonical document space to first reduce geometric variability and subsequently perform restoration under a normalized spatial layout. This formulation is particularly suitable for structured tax documents, where the relative positioning of fields, separators, tables, and encoded regions follows a rigid pattern. Restoration is not performed as a free-form transformation of the original degraded image but as a reconstruction conditioned on a template within a previously aligned geometry. This alignment stage constitutes the Digital Image Processing (DIP) component of the method and is performed prior to deep restoration.

Let $I_{noisy}\in [\mathrm{0,1}{]}^{H_0\times W_0}$ be the observed degraded image and $M_{master}\in [\mathrm{0,1}{]}^{H\times W}$ the master template in canonical space. To map $I_{noisy}$ into this space, a homography $H\in {\mathbb{R}}^{3\times 3}$ is estimated between the degraded document and the template. In the original formulation, this estimation is performed using SIFT keypoint detection on the degraded image, descriptor matching via BFMatcher with $L_2$ distance, correspondence filtering using Lowe’s ratio test, and robust transformation estimation via RANSAC. Once the homography is estimated, the degraded image is warped as:

$$I_{aligned}=\omega \left(H,I_{noisy}\right)$$

(3)

This operation projects the degraded image into the canonical coordinate system via a warp operator $\omega$, yielding the aligned representation that is used as input to the restoration block.

In the supervised setting, the clean reference $I_{gt}\in [\mathrm{0,1}{]}^{H\times W}$ is expressed in the same canonical space as $I_{aligned}$, enabling learning under explicit geometric correspondence between input and target. Additionally, the method uses $M_{master}$, defined in canonical space with values in $\{0,127,255\}$. From this mask, three auxiliary maps are derived:

$$\alpha ={\displaystyle \frac{M_{master}}{255}},M_c=1\left[M_{master}=255\right],M_v=1\left[M_{master}\ne 127\right].$$

(4)

From this mask, the alpha map $\alpha$, a content mask $M_c$, defined on pixels with value 255, and a validity mask $M_v$, defined on pixels different from 127 are obtained. These maps are not direct inputs to the backbone but serve as guidance signals that control structural fusion and the loss function.

The effective model input is constructed by concatenating the aligned degraded image and the canonical template $I_{template}$, forming a two-channel tensor:

$$X=\mathrm{concat}\left(I_{aligned},I_{template}\right)\in {\mathbb{R}}^{2\times H\times W}$$

(5)

In this way, the network learns to restore degraded content explicitly conditioned on the expected document structure. The template acts as the primary structural prior, while the mask is reserved for guiding permissible edits and region-specific optimization. This design combines observed visual information with structural prior knowledge consistent with the rigid nature of tax form layouts.

3.2.2. Model and Structure-Constrained Optimization

The restoration core is implemented using an encoder–decoder architecture based on a U-Net design, tailored for grayscale document restoration under template guidance. The encoder begins with a double-convolution block and proceeds through four stages of spatial downsampling. Each convolutional block consists of two $3\times 3$ convolutions, GroupNorm normalization, and ReLU activation, while spatial downsampling is performed using MaxPool2d. The decoder mirrors this structure using transposed convolutions for upsampling, skip connections from the encoder, and additional double-convolution blocks to refine reconstruction. The output layer consists of a $1\times 1$ convolution followed by a sigmoid activation, producing a single-channel prediction ${\widehat{I}}_{raw}\in [0,1{]}^{H\times W}$. Both during training and inference, the model is instantiated with $in\_channels=2$.

The raw network prediction is not directly used as the final output. Instead, the method applies a template-constrained fusion mechanism defined as:

$${\widehat{I}}_{fused}=\alpha \odot {\widehat{I}}_{raw}+\left(1-\alpha \right)\odot I_{template}$$

(6)

where $\odot$ denotes element-wise multiplication. This operation enforces that the network is allowed greater freedom only in regions enabled by the structural guide, while preserving the template in regions where modification is not desired. The alpha map $\alpha$ is used as a per-pixel blending weight between the network prediction and the canonical template. Specifically, background, fixed structure, and editable content regions are encoded in the master mask with values 0, 127, and 255, respectively. Consequently, $\alpha$ induces a differentiated constraint: background regions are primarily taken from the template, editable regions are primarily taken from the network prediction, and structural regions receive an intermediate blending value. This intermediate value implements a soft constraint rather than a hard copy of the template. Thus, structural pixels are biased toward the canonical template while still allowing smooth transitions with neighboring editable regions, avoiding abrupt visual discontinuities at field boundaries.

Model optimization is based on a composite loss function that integrates global fidelity, structural similarity, and content-specific reconstruction. First, a global $L_1$ term restricted to valid regions is computed on the fused output:

$${\mathcal{L}}_{global}=\mathrm{masked}{L}_1\left({\widehat{I}}_{fused},I_{gt},M_v\right).$$

(7)

Second, a structural loss based on SSIM is incorporated:

$${\mathcal{L}}_{ssim}=1-\mathrm{SSIM}\left({\widehat{I}}_{fused},I_{gt}\right).$$

(8)

Finally, an $L_1$ term focused exclusively on editable regions is computed on the raw prediction:

$${\mathcal{L}}_{content}=\mathrm{masked}{L}_1\left({\widehat{I}}_{raw},I_{gt},M_c\right).$$

(9)

The total objective function is defined as:

$$\mathcal{L}={\lambda }_1{\mathcal{L}}_{global}+{\lambda }_s{\mathcal{L}}_{ssim}+{\lambda }_c{\mathcal{L}}_{content}$$

(10)

where the weights ${\lambda }_1$, ${\lambda }_s$ and ${\lambda }_c$ control the contribution of each component. This formulation enables simultaneous enforcement of global document coherence, perceptual similarity of the final output, and localized reconstruction accuracy in regions where the model is expected to intervene. In general terms, ${\mathcal{L}}_{global}$ enforces similarity between the fused output and the clean reference, but only over valid regions defined by $M_v$. Since $M_v$, this term supervises background and editable content, but excludes fixed structural regions marked with value 127. Consequently, it does not directly penalize pixel-wise errors in structural zones. ${\mathcal{L}}_{\mathrm{ssim}}$ is computed over the full ${\widehat{I}}_{\mathrm{fused}}$ image and encourages local structural consistency using Gaussian windows of size 11 with $\sigma =1.5$. This term acts as a global constraint, preserving visual coherence and layout consistency even in regions not directly supervised by the $L_1$ term. The ${\mathcal{L}}_{\mathrm{content}}$ term plays a critical role. It is computed not on the fused output but on the raw prediction ${\widehat{I}}_{\mathrm{raw}}$, and only over editable regions defined by $M_c$. This enforces that the network learns to accurately reconstruct variable content in regions where modification is explicitly required. Figure 3 illustrates the complete pipeline.

3.2.3. Training and Inference Procedure

The DL component is trained in canonical space using supervised pairs of aligned degraded and clean images, together with the canonical template and the three guidance maps derived from the mask. To this end, two dataset construction strategies are implemented. During training, random patches are extracted from the aligned images, increasing the spatial variability observed by the model. During validation, deterministic coordinates are generated for each image in order to maintain a stable evaluation across epochs. In both cases, cropping is performed consistently over the degraded image, the clean image, the template, the alpha map, the content mask, and the validity mask, ensuring that all signals preserve exact spatial correspondence. The model input is formed by concatenating the degraded patch and the template patch, while the remaining maps are subsequently used in fusion and loss computation.

Training optimizes the model parameters using AdamW. At each iteration, the model produces a raw prediction from the input patch; this prediction is then fused with the template using the alpha map, the three loss terms described above are computed, and the parameters are updated through backpropagation. Validation follows exactly the same scheme. This procedure ensures that the network not only generates a reconstruction compatible with the clean reference but also does so under the structural constraints defined by the template and the mask.

During inference, full resolution aligned images are not processed in a single pass, but through an overlapping tiling strategy. For each document, the two-channel input tensor is first constructed from the degraded image and the template. This tensor is then divided into tiles of size $tile\_size$, with overlap controlled by $tile\_overlap$. Each tile is processed independently by the trained network, and the resulting predictions are recombined using a weight map based on a Hann window, with the aim of attenuating border artifacts between adjacent tiles. The aggregated prediction ${Î}_{raw}$ is finally fused with the template using the same $\alpha$-guided rule employed during training, producing the final restored output in canonical space (see Table 2).

3.3. Baselines and Evaluation Protocol

For the comparative evaluation, the proposed method was contrasted with representative baselines under a common experimental protocol in canonical space. The comparison included: (i) Noisy input, used as a no-restoration reference; (ii) CRP (classic restoration pipeline), a classical DIP-based restoration pipeline evaluated only at inference time; (iii) U-Net baseline [35], which performs supervised restoration from the aligned degraded image without incorporating a structural prior; (iv) U-Net + prior, which incorporates the canonical template as an additional input but does not apply the structurally constrained fusion or the full loss formulation of the proposed method; (v) DE-GAN [36] baseline; and (vi) an attention-based baseline inspired by Restormer [37]. For all learned baselines, the restoration stage was evaluated after geometric normalization, so that the comparison was performed under the same document geometry and with paired samples ($nois{y}_{aligned}$, $clea{n}_{aligned}$). This protocol constitutes a conservative evaluation of the proposed framework, since all learned baselines receive previously aligned inputs. Consequently, the comparison isolates the restoration component and prevents possible failures in geometric normalization from dominating the results.

All restoration models were evaluated on a curated and aligned subset of 2000 synthetic tax documents, divided into 1600 images for training, 200 for validation, and 200 for testing. A common training budget was imposed for all learned methods, consisting of 50 epochs and 500 steps per epoch, without early stopping, for a total of 25,000 optimizer updates per model. Training was performed through random patch sampling. U-Net, U-Net + prior, and the proposed model were trained with $256\times 256$ patches and a batch size of 8; DE-GAN used $128\times 128$ patches with a batch size of 16; and the attention-based baseline used $128\times 128$ patches with a batch size of 4. All experiments were run on a workstation equipped with an Intel Xeon Gold 6230R processor, 250 GB of RAM, and two NVIDIA RTX A6000 GPUs with 48 GB of memory each. Under this protocol, geometric normalization was treated as a fixed preprocessing stage and was evaluated separately before analyzing restoration performance.

The evaluation combined image-level fidelity metrics, binary quality metrics in the filled-in regions, and OCR-based functional metrics. Let $I_{gt}$ be the clean reference image and $Î$ the restored image, both in grayscale and in canonical space. Radiometric fidelity was quantified using the mean squared error and was computed as follows:

$$MSE={\displaystyle \frac{1}{HW}}{\displaystyle \sum _{u=1}^H}{\displaystyle \sum _{v=1}^W}{\left(\widehat{I}\left(u,v\right)-I_{gt}(u,v)\right)}^2$$

(11)

from which the Peak Signal-to-Noise Ratio (PSNR) was computed as:

$$\mathrm{P}\mathrm{S}\mathrm{N}\mathrm{R}=10{\log }_{10}\left({\displaystyle \frac{{255}^2}{\mathrm{M}\mathrm{S}\mathrm{E}}}\right).$$

(12)

Structural similarity was evaluated using the Structural Similarity Index (SSIM),

$$\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}\left(\widehat{I},I_{gt}\right)={\displaystyle \frac{\left(2{\mu }_{\widehat{I}}{\mu }_{I_{gt}}+c_1\left)\right(2{\sigma }_{\widehat{I}{I}_{gt}}+c_2\right)}{\left({\mu }_{\widehat{I}}^2+{\mu }_{I_{gt}}^2+c_1\left)\right({\sigma }_{\widehat{I}}^2+{\sigma }_{I_{gt}}^2+c_2\right)}}$$

(13)

where $\mu$, ${\sigma }^2$ and ${\sigma }_{\widehat{I}{I}_{gt}}$ represent local means, variances, and covariance, respectively.

For the binarization metrics, both $I_{gt}$ and $Î$ were converted into binary foreground masks using either an adaptive or global threshold, depending on the selected method. In general terms, the binary mask $B$ was defined as:

$$B\left(u,v\right)=\{\begin{array}{cc}1,& I\left(u,v\right)<\tau (u,v),\\ 0,& \mathrm{otherwise},\end{array}$$

(14)

where $\tau (u,v)$ represents the binarization threshold. The binary metrics were computed over a region of interest $\Omega$ corresponding to the filled-in fields of the document, that is, for pixels $(u,v)\in \Omega$. Over this region, the following quantities were defined:

$$\begin{gathered} TP=\left|\left\{B_{\widehat{I}}\right(u,v)=1\mathrm{and}{B}_c(u,v)=1\}\right| \\ FP=\left|\left\{B_{\widehat{I}}\right(u,v)=1\mathrm{and}{B}_c(u,v)=0\}\right| \\ FN=\left|\{B_{\widehat{I}}\right(u,v)=0\mathrm{and}{B}_c(u,v)=1\}| \end{gathered}$$

(15)

where $\left|.\right|$ denotes the cardinality of the set, that is, the number of pixels satisfying the condition. From these quantities, Precision, Recall, and F-measure were computed as:

$$P={\displaystyle \frac{TP}{TP+FP}}, \mathrm{R}={\displaystyle \frac{TP}{TP+FN}}, F-\mathrm{measure}={\displaystyle \frac{2PR}{P+R}}$$

(16)

Binary distortion was also evaluated using the Distance Reciprocal Distortion (DRD). Let $E=\left\{\left(u,v\right)\in \mathsf{\Omega }:B_{\widehat{I}}\left(u,v\right)\ne B_c\left(u,v\right)\right\}$, be the set of erroneous pixels. The implementation used computes:

$$\mathrm{D}\mathrm{R}\mathrm{D}={\displaystyle \frac{1}{NUBN}}\sum _{\left(u,v\right)\in E}DRD\left(u,v\right),$$

(17)

where $NUBN$ is the number of non-uniform $8\times 8$ blocks in the binary reference within the evaluated region. For each erroneous pixel $(u,v)$, DRD is computed as follows:

$$DRD\left(u,v\right)={\displaystyle \sum _{i=-2}^2}{\displaystyle \sum _{j=-2}^2}\left|B_c\left(u+i,v+j\right)-B_{\widehat{I}}\left(u,v\right)\right|\mathcal{W}\left(i,j\right),$$

(18)

with a normalized weight mask:

$$\mathcal{W}={\displaystyle \frac{1}{16}}\left[\begin{array}{ccccc}0& 0& 1& 0& 0\\ 0& 1& 2& 1& 0\\ 1& 2& 0& 2& 1\\ 0& 1& 2& 1& 0\\ 0& 0& 1& 0& 0\end{array}\right].$$

(19)

Finally, functional utility was measured using OCR applied to the identification fields delimited by the PDF template. Text recognition was performed with the Tesseract OCR engine version 5.5.1, compiled with Leptonica 1.82.0. It was configured with page segmentation mode (PSM) 6, corresponding to a uniform text block, and the Spanish language model (spa), without additional preprocessing of the restored image. For each sample $n$, the seven fields of the identification section were automatically extracted (tipo_doc, primer_apellido, segundo_apellido, primer_nombre, segundo_nombre, razon_social, nombre_comercial). The ground-truth texts were then concatenated into a string $t_n$, and the OCR-recognized texts into a string ${\widehat{t}}_n$. Before computation, both strings were normalized by converting them to uppercase, removing accents, and collapsing multiple spaces. With this normalization, the character and word error rates for each sample were defined as follows:

$${\mathrm{C}\mathrm{E}\mathrm{R}}_n={\displaystyle \frac{E{D}_{\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{r}}\left({\widehat{t}}_n,t_n\right)}{\max (1,{\left|t_n\right|}_{\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{r}})}},{\mathrm{W}\mathrm{E}\mathrm{R}}_n={\displaystyle \frac{E{D}_{\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}}\left({\widehat{t}}_n,t_n\right)}{\max (1,{\left|t_n\right|}_{\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}})}},$$

(20)

where $E{D}_{\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{r}}$ and $E{D}_{\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}}$ correspond to the Levenshtein distance at the character and word levels, respectively. Micro-averages were used for reporting the results:

$${\mathrm{C}\mathrm{E}\mathrm{R}}_{\mathrm{m}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}}={\displaystyle \frac{{\sum }_{n=1}^N{ED}_{\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{r}}({\widehat{t}}_n,t_n)}{{\sum }_{n=1}^N{\left|t_n\right|}_{\mathrm{c}\mathrm{h}\mathrm{a}\mathrm{r}}}},{\mathrm{W}\mathrm{E}\mathrm{R}}_{\mathrm{m}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{o}}={\displaystyle \frac{{\sum }_{n=1}^N{ED}_{\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}}({\widehat{t}}_n,t_n)}{{\sum }_{n=1}^N{\left|t_n\right|}_{\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{d}}}}.$$

(21)

4. Results

4.1. Validation of the Synthetic Dataset

4.1.1. Partition Homogeneity

The synthetic dataset was split into 80%/10%/10% for training, validation, and testing, respectively. To rule out partitioning biases in the degradation variables, homogeneity across partitions was evaluated.

Table 3 presents the homogeneity verification across dataset partitions for the degradation variables: contrast $\alpha$, brightness $\beta$, angle $\theta$, and translations $t_x$ and $t_y$. The Kullback-Leibler (KL) divergence is reported as a descriptive measure of discrepancy, and the p-value of the energy distance test is used as an inferential contrast of the null hypothesis $H_0$: both partitions come from the same distribution. In all comparisons (train-validation and train-test), the p-values were greater than $0.05$; therefore, there is no statistical evidence to reject $H_0$. Consequently, the partitions maintain similar degradation distributions, reducing the risk of experimental bias due to partitioning and supporting the attribution of observed performance differences to the model rather than to systematic imbalances in the degradation parameters.

4.1.2. Validity Against Real Data Using a Coverage Criterion

Due to privacy restrictions, the available real dataset does not correspond to a designed probabilistic sample. In addition, with approximately 37 real documents, the power of two-sample tests is limited, and several image metrics do not follow normal distributions, reducing the usefulness of parametric tests in this setting. Consequently, a coverage-based compatibility validation was adopted. The objective was to verify that the real documents were not atypical with respect to a high-density region induced by a large synthetic sample, using PCA fitted on the synthetic data and an elliptical region defined by the threshold ${\mathsf{\chi }}_{\mathrm{k}}^2\left(0.95\right)$.

Using 1000 synthetic samples and 37 real documents, a coverage of 35/37 (94.6%) was obtained, with a minimum 95% confidence bound of $81.8\%$, using $k=4$ principal components and a threshold of $\chi 2=9.488$ with Ledoit-Wolf covariance. This result supports that the generator produces samples consistent with the available real set under the defined compatibility criterion (see Figure 4).

4.2. Validation of the Restoration Method

4.2.1. Performance of Geometric Normalization

The geometric normalization stage was evaluated on the test partition using robust statistics, specifically the median and interquartile range (IQR), in order to characterize variability across samples. First, the alignment success rate reached 100% (200/200 test documents), defined as the proportion of samples for which a valid homography was estimated and an aligned output was correctly generated. In addition, considering the full aligned and curated dataset, all $2000/2000$ selected samples were successfully normalized, with no failures recorded during dataset construction.

Second, the inlier ratio was quantified, defined as $num\_inliers/good\_matches$, which directly reflects the geometric consistency of the correspondences retained by RANSAC. In the test set, this ratio had a median of 0.7958 and an IQR from 0.7499 to 0.8213. Complementarily, the number of inliers showed a median of 410.5 (IQR: 317.0–518.25), while the number of retained valid matches reached a median of 524.5 (IQR: 411.0–631.0) (see Table 4). Figure 5 shows an example of the result on a test-set case.

4.2.2. Restoration Results and OCR Utility

Figure 6 and Table 5 summarize the qualitative and quantitative comparison between the proposed method and the considered baselines, including a no-restoration reference (Noisy input) and a classical restoration pipeline (CRP). The results in Table 5 correspond to the fixed-budget scenario, in which all learned models were trained for 50 epochs and 500 steps per epoch, ensuring a controlled comparison across architectures. The evaluation was performed on the test set aligned in canonical space, allowing all methods to be compared under the same geometry. Therefore, the results in Table 5 should be interpreted as a comparison focused on the restoration stage, under favorable and homogeneous conditions for all learned models. In this scenario, PSNR and SSIM quantify radiometric and structural fidelity with respect to the clean reference, while F-measure and DRD evaluate the preservation of the document foreground after binarization. To measure functional utility, CER and WER were computed from OCR on the identification region of the form, comparing the recognized text with the synthetic reference content available in the dataset JSON files. In particular, these metrics were aggregated as micro-averages, in order to reflect the total character and word error over the full evaluated set.

First, the results show that both Noisy input and CRP remain clearly below the learned models. Although CRP slightly improves over the degraded input image (PSNR: 15.4991 vs. 14.5045; CER: 0.1455 vs. 0.1538), its performance remains limited across all metrics, confirming that a classical pipeline is insufficient under complex degradations in structured tax documents. In contrast, all deep learning-based methods produce a substantial improvement in visual fidelity, structural consistency, and OCR legibility.

Among the learned baselines, U-Net + prior achieved the best performance in the visual and structural metrics, reaching the highest PSNR (28.3218 ± 2.3118), the highest SSIM (0.9841 ± 0.0079), the best F-measure (0.9243 ± 0.0261), and the lowest DRD (183.1854 ± 97.3021). This result indicates that, once the geometry is fixed through prealignment, explicitly incorporating the canonical format as a prior provides a clear advantage over purely end-to-end restoration. Indeed, U-Net + prior outperforms both the U-Net baseline and the Attention baseline, suggesting that, in this problem, explicit document structure is more beneficial than only increasing architectural complexity or introducing attention mechanisms without geometric constraints.

However, when the comparison shifts to the functional level, the most relevant behavior corresponds to the proposed hybrid method. Although it does not achieve the best PSNR or SSIM, it obtains the best OCR utility results, with the lowest CER (0.1091) and the lowest WER (0.3495) in the entire comparison. This is particularly important because the objective of the problem is not only to restore the global appearance of the form, but also to correctly preserve and recover the variable content inserted into a rigid template. In this sense, the hybrid method shows a better balance between visual restoration and semantic text recovery, which is consistent with its design: the canonical template resolves the fixed structure, while the network focuses on reconstructing useful content within the informative regions.

On the other hand, the DE-GAN baseline showed the worst performance among the learned models (CER: 0.3928; WER: 0.9353), even with structural results below several non-adversarial baselines. This behavior suggests that, for this type of form, adversarial optimization does not offer a clear advantage and may even affect the stability of the restoration. Similarly, the Attention baseline improves over Noisy input and CRP, but remains below the prior-guided variants, reinforcing the central hypothesis of this work: in documents with a rigid layout, learned attention alone does not replace the explicit incorporation of structural information.

This allows two main conclusions to be drawn. First, canonical prealignment followed by supervised restoration constitutes an effective strategy for this problem. Second, the best visual quality does not necessarily coincide with the best OCR utility: while U-Net + prior dominates in fidelity and structural preservation, the proposed hybrid method provides the best textual recovery, which is ultimately the most relevant property in automatic reading and document-processing scenarios.

As a complementary analysis, Table 6 reports a convergence-oriented scenario based on early stopping. Table 5 should be interpreted as the main comparison under a fixed and homogeneous training budget, whereas Table 6 presents a convergence-based scenario in which each learned model is evaluated at the best checkpoint determined by the evolution of the validation criterion. In this configuration, early stopping was applied with a patience of 5 epochs and a maximum of 120 epochs, while keeping the same dataset, validation split, and 500 steps per epoch used in the fixed-budget setting. The results show that this configuration preserves the general ranking trends: U-Net + prior remains the strongest method in terms of visual and structural fidelity, whereas the proposed hybrid model achieves the best OCR-oriented performance (CER = 0.1072, WER = 0.3175). This result suggests that, when training is allowed to adapt to convergence, the hybrid formulation better supports the objective of improving the functional legibility of the document, rather than optimizing only global image similarity.

Figure 7. Improvement in validation metric (from epoch 1) over training epochs under early stopping.

Figure 7. Improvement in validation metric (from epoch 1) over training epochs under early stopping.

4.2.3. Computational Cost and Practical Efficiency

Table 7 compares the computational cost of the evaluated methods in terms of number of parameters, arithmetic complexity, peak GPU memory, training time, and inference latency. This analysis complements the restoration and OCR results, since in real document-processing applications, not only output quality matters, but also the practical feasibility of the method under time and resource constraints.

First, the U-Net-based models show a very similar computational profile. The U-Net baseline, U-Net + prior, and the proposed hybrid model have around 7.76 M parameters and 27.3 G FLOPs, with memory requirements close to 1.5 GB and training times per epoch between 74 and 77 s. This indicates that incorporating the canonical prior and the hybrid scheme does not introduce a severe computational penalty compared with the base model. In particular, using the 1600 training images, the proposed method requires 77.02 ± 0.69 s per epoch and 64.18 ± 0.57 min to complete 50 epochs, representing only a moderate increase compared with the U-Net baseline (61.82 ± 0.40 min) and U-Net + prior (62.97 ± 1.12 min).

The DE-GAN baseline has the lowest number of parameters (1.810 M) and the lowest theoretical complexity (0.428/0.214 G FLOPs/MACs), as well as the lowest inference latency among the learned models (2.67 ± 0.12 ms). However, this cost advantage does not translate into better restoration quality, since it was one of the worst-performing methods in the previously reported metrics. This suggests that low arithmetic complexity alone is not sufficient to adequately restore degraded rigid forms.

In contrast, the Attention baseline represents the least favorable case in terms of practical efficiency. Although its number of parameters (3.293 M) is lower than that of the U-Net variants, its peak memory reaches 3.188 ± 0.019 GB, more than twice that of the other learned models, and its inference latency reaches 20.61 ± 0.73 ms, well above the rest. It also records the highest training time per epoch (91.14 ± 0.33 s). Consequently, this model does not offer an attractive cost-benefit ratio for real deployment scenarios.

Finally, CRP does not require training, but its inference latency (132.17 ± 9.21 ms) is the highest among all evaluated methods. This indicates that avoiding training cost does not necessarily imply greater operational efficiency when large volumes of documents must be processed.

The proposed hybrid method offers a favorable trade-off between performance and computational cost. Although it is not the lightest method, it maintains memory requirements, training time, and latency very close to U-Net + prior, while improving the final utility for OCR. From a practical perspective, this suggests that the functional gain of the hybrid approach is achieved without introducing prohibitive computational overhead.

4.2.4. Component Ablation

Table 8 presents the results of the ablation study of the proposed method on the test set, with the aim of isolating the effect of three key components: the content-oriented masked loss, the structural fusion with the canonical template, and the explicit visual template prior. In all cases, the variants were retrained under the same experimental protocol as the full model, so that the observed differences could be attributed primarily to the removed component.

The most consistent result corresponds to the w/o template prior variant, which produces the largest overall degradation in the visual and structural metrics. Compared with Hybrid full, this ablation reduces PSNR from 26.4961 to 24.2217, SSIM from 0.9607 to 0.9438, and F-measure from 0.9113 to 0.8701, while increasing DRD from 297.3505 to 325.9923. This behavior indicates that the explicit canonical prior is a central component of the method, since it provides a stable structural reference that improves restoration in documents with a fixed layout. In other words, when the model loses access to the visual template of the form, its ability to preserve the global organization of the document is clearly reduced.

The other two ablations show a more subtle behavior. Both w/o masked content loss and w/o template fusion obtain higher values than the full model in PSNR, SSIM, F-measure, and DRD. In particular, w/o template fusion achieves the best visual results in the table, with PSNR = 28.3505 ± 2.3779, SSIM = 0.9827 ± 0.0082, and DRD = 146.0933 ± 90.4275. However, this improvement does not translate into better textual utility: its OCR metrics (CER = 0.1124, WER = 0.3588) remain below those of the full model. A similar pattern is observed for w/o masked content loss, which improves the visual restoration metrics but worsens text recovery (CER = 0.1142, WER = 0.3627).

This result is particularly relevant for interpreting the behavior of the proposed method. The full variant is not optimized solely to maximize global image similarity, but rather to correctly recover the variable content of the form. The masked loss and template fusion appear to act as functional regularization mechanisms, sacrificing part of the global visual fidelity to favor the preservation and legibility of useful content. Therefore, although the simplified variants may produce images closer to the GT in terms of PSNR/SSIM, the full model remains the one with the best OCR performance, achieving the lowest CER (0.1091) and WER (0.3495) in the ablation study.

Therefore, first, the template prior is the most critical component for maintaining the structural quality of the method. Second, the combination of content-oriented loss and template fusion does not necessarily maximize classical visual metrics, but it does improve the final utility of the restored document for automatic reading.

4.3. Functional Evaluation on Real Documents

To complement the controlled evaluation on synthetic data, a functional test was conducted on the 37 available real documents. Unlike the synthetic set, these documents do not have a clean reference image that would allow pixel-level metrics such as PSNR or SSIM to be computed. For this reason, the evaluation on real data was not designed as a direct measurement of visual fidelity, but rather as a test of textual recoverability after restoration.

For each real document, two versions were considered: the original degraded image and the image restored with the proposed hybrid method. In addition, a textual ground truth was manually constructed from the full form page, including both fixed template text and the information filled into the document fields.

The evaluation was performed using three independent OCR engines: GLM OCR [38], Amazon Textract [39], and Tesseract [40]. For each engine, text was extracted from the full page in both image versions. The percentage of ground-truth words recovered by the OCR output was then measured. This metric is interpreted as lexical coverage of the document, not as full OCR accuracy, since it does not explicitly penalize insertions, false positives, or semantic field-level errors. Formally, for each OCR engine, coverage was computed as:

$$Coverage={\displaystyle \frac{\left|W_{gt}\cap W_{ocr}\right|}{\left|W_{gt}\right|}}$$

(22)

where $W_{gt}$ represents the list of normalized ground-truth words and $W_{ocr}$ the list of normalized words detected by the OCR engine. Coverage was computed as lexical recovery, counting each ground-truth word as recovered if it appeared at least once in the OCR output.

Table 9 presents the results obtained. For the three OCR engines evaluated, restoration increased the coverage of ground-truth words compared with the degraded real image.

Because coverage is computed over the full page, the metric combines fixed template text and filled-in content; therefore, this test evaluates the document’s overall textual recoverability. The results show that the proposed restoration improves textual recoverability in real documents across the three OCR engines considered. The gain is moderate for GLM OCR, which already achieves high coverage on the original degraded image, and more pronounced for Amazon Textract and Tesseract. In particular, Tesseract shows the largest absolute improvement, increasing from 11.84% to 29.89%, suggesting that OCR engines more sensitive to noise, misalignment, and visual degradation may benefit more clearly from structure-guided restoration.

This test does not replace the main quantitative evaluation performed on synthetic data, where a clean reference and experimental control are available. However, it provides complementary evidence on real documents, showing that the proposed method not only improves visual and structural metrics on synthetic data, but also increases full-page textual recovery in real scenarios.

5. Discussion

The results suggest that, for structured tax documents, restoration evaluation should not focus solely on the generative capacity of increasingly large image models, but also on their ability to recover the document without introducing uncertainty into legally sensitive content. In this type of document, a spurious stroke, an altered digit, or a displaced field boundary can compromise the functional validity of the record. This is compounded by an equally relevant practical constraint: such documents often contain confidential information, so their processing with large-scale models, especially when they rely on external infrastructure or remote services, introduces additional requirements for security, traceability, and data governance. Therefore, constraining restoration through canonical alignment, templates, and masks is not only a design decision, but also a way to reduce the degrees of freedom in a problem where structural fidelity, process control, and information protection are more important than isolated perceptual quality.

In this direction, the findings reinforce a broader idea: end-to-end deep learning is not always the most suitable option when the domain imposes strong geometric and semantic constraints. In the proposed approach, the classical DIP component does not compete with the network; instead, it simplifies the problem by first fixing the geometry and providing an explicit layout prior. This helps explain why template-guided variants showed advantages over adversarial alternatives or attention-only models. In particular, the hybrid method obtained the lowest CER and WER values in the evaluated protocol, supporting the usefulness of combining geometric normalization, structural priors, and supervised restoration when the final objective is to improve automatic reading. At the same time, recovering small letters, fine separators, and few-pixel-wide lines remains difficult for any method, because these are precisely the structures most vulnerable to degradation and local ambiguity.

Regarding the scope of the study, the quantitative comparison was conducted under a homogeneous and controlled protocol, in order to isolate the effect of the restoration stage and compare the methods under the same document geometry. For this reason, the main results should be understood as a specific evaluation of restoration in canonical space, with previously aligned inputs. The evaluation on real data, in turn, measures textual recoverability rather than pixel-level restoration fidelity, since no clean visual reference is available for those documents. Consequently, the results on real documents serve as complementary functional evidence: they show that the proposed restoration increases OCR coverage under real conditions, but they are not intended to replace the controlled evaluation performed on synthetic data. Likewise, the objective was not to propose an OCR system, but to improve the functionality of the OCR engine used after restoration. Therefore, CER, WER, and word coverage should be interpreted as OCR-dependent functional metrics, useful for characterizing the practical utility of the restored document.

Finally, the use of synthetic data remains the main factor requiring caution in the interpretation of the results. Although the coverage-based validation showed compatibility between the generator and the available real sample, with 35 out of 37 real documents falling within the confidence region induced by the synthetic data, this does not rule out the existence of extreme cases that are not yet represented. This consideration is especially important in the tax and legal context, where the problem is not only to recover content, but also to avoid introducing plausible but false content. In this scenario, structural control of the restoration process and the explicit reduction of degrees of freedom are central elements of the proposed approach. As a natural continuation of this work, it is pertinent to extend the validation with more diverse real documents, study out-of-distribution failures, and incorporate uncertainty estimation mechanisms before considering operational use in sensitive environments.

6. Conclusions

This work addressed the restoration of degraded tax forms as a recovery problem under structural constraints, rather than as an unconstrained generation task. The combination of geometric normalization, a canonical template, structural masks, and supervised restoration reduced the model’s degrees of freedom and guided the restoration process toward functionally relevant regions of the document.

The results show that, under a controlled synthetic protocol and in canonical space, template-guided methods provide advantages over baselines without structural information. U-Net + prior achieved the best visual and structural fidelity, whereas the proposed hybrid method obtained the lowest CER and WER values in the OCR evaluation. This discrepancy between visual and functional metrics reinforces the need to evaluate document restoration not only in terms of image similarity, but also in terms of usefulness for automatic reading.

As a complementary evaluation, real documents showed improvements in OCR coverage after restoration. However, these results should be interpreted with caution, since no clean visual reference is available and the coverage metric includes both fixed template text and filled-in content. A priority direction for future work is to extend the evaluation to a larger set of real documents, separate fixed and variable fields, evaluate the full end-to-end pipeline, and incorporate uncertainty estimation before operational use in tax or legal contexts.