A Single Character Based Embedding Feature Aggregation Using Cross-Attention for Scene Text Super-Resolution

1. Introduction

As a vision challenge task, scene text image super-resolution (STISR) aims to reconstruct high-resolution (HR) textual sequences from low-resolution (LR) text contained images. Due to imaging conditions and other reasons, scene text images often suffer from defects such as low resolution, which greatly affects the accurate acquisition of text details as well as subsequent recognition tasks, such as scene text recognition [1,2,3] and scene text detection [4,5]. Existing methods typically utilize text-specific features, such as stroke and character structure, to improve the comprehensibility and discrimination of the resulting super-resolution text images [6,7,8]. In this paper, we focuses on relieving the limitations as the dense distribution of characters and the inefficient representation of foreground sequences from complex background in real scenes.

With the bloom of deep learning, Deep Convolutional Neural Network(DCNN) has become the basic component for STISR due to its powerful nonlinear mapping capability and adaptability. TextSR [6] uses recognition loss to guide the training of generative adversarial networks. TPGSR [9] incorporates semantic features extracted by a text recognizer into the generative network. TATT [10] devised a TP interpreter and structural consistency loss for processing spatially deformed text. As shown in Figure 1(a,b), the above baseline does not adequately consider the inherent dense distribution of characters and the detrimental effect of complex backgrounds on the local representation and reconstruction of characters.

Afterwards, TSRN [7] proposed to capture contextual information in images by combining sequential residual blocks. In addition, TBSRN [8] exploits the advantage of the attention module in processing sequential information, which makes the model robust to text regions in any orientation. Whereas the design of Gestalt [11] focuses on modelling the internal structure at the character stroke level in text images. MTSR [12] makes use of multiple transformers to perform the multi-tasks of image reconstruction and super-resolution. These schemes usually use traditional convolution operations. Due to their limited receptive fields, they focus on capturing local features such as strokes and edges of characters.However, as continuous data, the readability of text also depends on global information such as the relative position and overall arrangement of characters. Therefore, a competent architecture that takes into account both local details and global semantic information needs to be further explored.

In recent years, MNTSR [13] using self-supervised end-to-end memory networks has been proposed, while PerMR [14] fuses low-level strokes and high-level semantics extracted from a text recognition network to improve the visual quality, and furthermore, TLWSR [15] proposes a weakly-supervised framework to implement STISR when only some of the HR labels are available. As shown in Figure 1(c), these methods use semantic information or only focus on the text structure, while ignoring the color and texture of the original image. Which leads to frequent color drift between the text area and the background area in the generated image.

Aiming at the above bottlenecks, this paper proposes a single character based embedding feature aggregation using cross-attention for scene text super-resolution tasks, with the following main contributions:

• This study proposes a two-branch feature aggregation strategy. It aggregates independently clipped single-character image features with corresponding character probability sequences. In this way, the high-level prior of the aggregation focuses on the structural details of the characters themselves, effectively reducing the interference of the complex background. In addition, the influence among neighbouring densely distributed characters is significantly reduced.
• Considering the harmonization capability of different sized convolutional kernels due to their specific receptive fields, a improved Inception module is introduced into the shallow layers, in order to apply the dynamic multi-scale feature extraction strategy. By dynamically weighting scaled convolutional kernels, the global overview features and fine-grained features are adaptively adjusted for each input, thus enriching the feature expressions to comprehensively understand the salient vision content.
• Using the idea that adaptive normalisation can learn the mapping relationship between different image domains, a colour correction operation is applied to adaptively adjust the mean and standard deviation of the pixels of target images. This improves the quality and effect of super-resolution reconstruction while keeping the content of the original image unchanged. Experiments are performed on the public dataset TextZoom, and the results show the superiority of the proposed model with the existing baselines. The average recognition accuracy on the test sets of CRNN, MORAN, and ASTER is improved by 1%, 1.5%, and 0.9%, respectively.

2. Related Works

2.1. Image Super Resolution

The super-resolution (SR) tasks aim to generate HR images from LR inputs through a priori spatial learning methodology. Traditional SR algorithms can be categorized as interpolation-based [16,17,18], frequency-domain-based [19,20] and learning-based [21] organizing by proposed order. With the rise of deep learning, Dong et al [22]. firstly suggested using a set of convolution layers to learn prior representations for super-resolution reconstruction. Since then, the scale of SR networks has been expended to be further implemented on STISR tasks. For instance, VDSR [23] as a SR architecture using 20 convolutional layers, ESPCN [24] formulated by a sub-pixel convolutional network, recursively structured deep network [25] and Deep residual network [26] with jump connections. After that, the SRGAN approach has adopted adversarial learning to the field of SR reconstruction. Recently, diffusion models have been further introduced to different vision tasks including SR after surpassing GANs. For instance, SRDiff [27] is the first diffusion-based SR model for single image reconstruction, which can effectively relieve the problems of spatial detail confusing and training pattern collapse in order to formulate realistic results.

2.2. Scene Text Recognition for STISR

Scene text recognition (STR) refers to the process of automatically detecting and recognizing text information in natural scene images. With the rise of deep learning, STR has made breakthrough progress. CRNN [2] transforms a text recognition problem into a sequence learning problem, modeling the position and shape of individual characters. MORAN [3] enables text detection and recognition through a spatial transformation network and a recognition network. Adaptive correction network and attention sequence recognition model introduced by ASTER [28]. TextSR [6], TPGSR [9], TATT [10] and other methods all use the above recognizer to extract text a priori to guide SR networks. In this paper, we generally apply the above three recognizers to capture semantic information and then allocate it to corresponding image features to improve reconstruction performance.

2.3. Scene Text Super Resolution

Different from Single Image Super-Resolution (SISR), STISR focuses on upgrading the quality of text images in order to improve the readability of the text and enhance downstream recognition tasks. In spite of the SISR approaches can theoretically be applied to STISR, e.g., Dong et al. perform the STISR task by extending the SRCNN [29] baseline to text images. Also, PlugNet [30] employs a lightweight SR module to extract features of LR images. However, STISR generally requires special processing to maintain the character and clarity of text. In recent years, more models specifically designed for STISR have been suggested. These schemes often utilize specific features of the text, such as stroke information and text structure, to improve the super-resolution results.

In addition, TextSR employs text recognition loss and text perception loss to preserve text feature, such as stroke clarity and word spacing, to ensure that super-resolution processed text can be recognized more accurately by optical character recognition (OCR) systems. To improve the effectiveness of STISR on real scenes, Wang et al. built a real scene text image dataset as TextZoom, and proposed TSRN using a centre-alignment module and sequence residual blocks to extract semantic information. Also, PCAN [31] adopts a parallel contextual attention network with specific mechanisms to effectively fuse multi-scale features and enhance the recovery of textual details. TATT then uses a TP interpreter module based on an attention architecture that aligns textual priori and image features based on a global attention. Moreover, C3-STISR [32] jointly utilizes recognition feedback, visual and linguistic multiple cues to enhance textual image and design triple cross-modal extraction and aggregation mechanisms for comprehensive guidance. Although the above methods focus on textual and characters reconstruction with prior details, the challenge still comes from the extreme dense text structure as well as the confusion with complex backgrounds.

3. The Proposed Network Architecture

The feedforward network structure for STISR is an important factor that affects the reconstructed processing of textual images. In this paper, the single character embedding feature aggregation for driving the cross-attention based STISR network (SCE-STISR) is shown as Figure 2. According to it, LR images $X_{LR}$ are used as inputs to the preprocessing, that employs a dynamic inception feature extraction to extract shallow features $X_I$, which also takes into account both global and local representations. Then a pre-trained text recognizer predicts text probability sequences $t_p$. Single character boundary detection clip $X_I$ into character feature $X_S$ according to character position information. $X_I$, $X_S$ and $t_p$ parallel input dual branch feature aggregation to guide the aggregation of visual and semantic representations. Ultimately, a high-level prior $P_2$ = $f_{DBFA}{(X}_I,X_S,t_p)$ is formulated as input of the reconstruction module, which includes a colour correction layer for improving colour consistency and accuracy, and finally a sequence residual block for performing SR image reconstruction.

3.1. Image Pre-Processing

Image pre-processing procedure consists of three parts. That is dynamic inception feature extraction, single character boundary detection and Text Recognizer. According to them, LR image ${\mathrm{X}}_{\mathrm{L}\mathrm{R}}$ are fed as inputs, and shallow features ${\mathrm{X}}_{\mathrm{I}}$, character image features ${\mathrm{X}}_{\mathrm{S}}$ and text priors ${\mathrm{t}}_{\mathrm{p}}$ are estimated as outputs.

3.1.1. Dynamic Inception Feature Extraction

In general, the receptive fields of a single convolution layer are limited, and only single-scale features can be extracted. However, the characters in text images has multi-scale property involving font, size, style, etc., which results in the inefficient of single convolution layer to adequately extract rich prior representations. The block of dynamic inception feature extraction(DIFE) [33] applied in this paper is shown as Fig.3. We use multi-scale convolution kernels to flexibly capture structural features at all levels in the text, from letter shape to word layout, to adapt to different datasets and task requirements. Then the 1x1 convolutional kernel is applied for dimensionality reduction or upgrading of feature channels, which adjusts the parameters scale and the network nonlinearity to formulate efficient architecture.

$$X_i=f_c(k^{h\times h},X_{LR}),h=\mathrm{1,3},\mathrm{5,7},9$$

(1)

where $X_{LR}\in {\mathcal{R}}^{H\times W\times 3}$ is LR text image, where H and W are the height and width, respectively. $X_i\in {\mathcal{R}}^{H\times W\times C}$ denotes the shallow features extracted from the text image at each layer with $iϵ \left[\mathrm{0,4}\right]$, and C denotes the number of feature channels, $k^{h\times h}$ as the size of the convolution kernel sized by $h\times h$, and $f_c(·)$ denotes the convolution function.

Since different inputs have different priorities in the feature space captured by multi-scale convolution kernels, therefore this paper introduce a dynamic feature weighting mechanism.This mechanism can adaptively assign specific weights to each kernel branch based on the specific feature distribution according to the input data. This strategy not only enhances the network's sensitivity and ability to capture diverse features, but also enables it to more accurately understand and process complex text structures, thereby improving the overall performance and generalization capabilities of the model. The computational process is represented as:

$$w_i=f_{sig}\circ f_{FC}\circ f_{GAP}\left(X_{LR}\right),i\in \left[\mathrm{0,4}\right]$$

(2)

$$X_I=\sum _{i=0}^4{w}_i{X}_i$$

(3)

where $\circ$ represents compound operation, $f_{GAP}$ means global average pooling operation on input $X_{LR}$. After that, through the $f_{FC}$ fully connected layer function, the respective weight coefficients of the five convolutions of different input samples are learned. Finally the weights on each dimension are constrained to [0,1] by sigmoid function $f_{sig}$ to obtain the normalized dynamic weights $w_i$. Also $X_I\in {\mathcal{R}}^{H\times W\times C}$ is the shallow feature output.

3.1.2. Single Character Boundary Detection

Since LR images usually contain multiple characters, the relative position and order of arrangement between characters is critical. Transformer's self-attention [34] can directly model the association between any two characters, thereby generating more accurate text position information. In order to effectively separate foregrounds from background and maximize textual priori information to guide semantic and visual reconstruction, this study adopts transformer-based single character of boundary detection (SCBD) [35] to predict the character positions for clipping individual characters as formulation of a image block sequence.

SCBD as shown in Figure 4, the input $X_I$ is flattened as $X_P=f_f\left(X_I\right)$, $X_P\in {\mathcal{R}}^{N\times (P^2\times C)}$ is a sequence of 2D image blocks, the resolution of each image block is $P\times P$, and the number of image blocks $N=H\times W/P^2$. To help the model maintain position sensitivity while processing the sequence data, a further position coding $E_{pos}$ is fed into the encoder to predict the character position:

$${\left\{\right(P_C^j,P_W^j\left)\right\}}_{j=0}^M=f_E(X_P+E_{pos})$$

(4)

In Equation (4), $(P_C^j,P_W^j)$ refers to the centre position and width of the each predicted character, j represents the jth character , as well as $f_E(·)$ refers to the cascade transformer encoder.

In order to reduce background interference, focus on the foreground of the text, and weaken the influence between neighbouring characters, we use the predicted positional information $(P_C^j,P_W^j)$ to crop out the corresponding single character features:

$$X_S^j=f_{clip}(X_I,(P_C^j,P_W^j\left)\right)$$

(5)

where $f_{clip}(·)$ denotes clipping operation, $X_S^j$ indicates a single character feature, $X_S^j$ is flattened as${\dot{X}}_S^j=f_f\left(X_S^j\right),$and$X_S=({\dot{X}}_S^0,{\dot{X}}_S^1,{\dot{X}}_S^2,...,{\dot{X}}_S^M)$represents the output of concatenation of ${\dot{X}}_S^j$.

3.1.3. Text Recognizer

The aim of text recognizer (TR) is to capture text probability sequences in LR images as a priori information, thus guiding the model to reconstruct SR images with precise text semantics. In the related works, the methods such as CRNN, Moran and Aster show excellent performance in the field of text recognition. Therefore, this paper selects these pre-trained recognition models to obtain the coding vector of the text category $t_p^j$:

$$t_p^j=f_{TR}\left(X_S^j\right)$$

(6)

In this equation, $f_{TR}(·)$ is the TR function and $t_p=(t_p^0,t_p^1,t_p^2,...,t_p^M)\in {\mathcal{R}}^{M\times S}$ is the probability sequence of the text prior, where M denotes the length of the sequence, and S as the number of categories in the reference text label set. In general, this set consists of the Arabic numerals 0-9, 26 letters of the alphabet, and a blank character.

3.2. Dual Branch Feature Aggregation

In the proposed architecture, dual branch feature aggregation (DBFA) is the key part.The purpose of DBFA is to interpret text priori $t_p$ before image features, thereby exerting precise semantic guidance on relevant spatial locations in the image feature domain. With the deep interaction between prior knowledge of the text and image features, character adhesion and background interference gradually intensify, leading to wrong semantic guidance. In order to address the above challenges, this study focuses on the aggregation of individual character features, while aggregating global features to supplement missing background information.

As shown in Figure 5, self-attention by capturing the semantic connections of characters and enhancing relevance, it helps understand the structure and meaning of the text. In this study, text priori $t_p$ is projected to C-channel using linear projection to match the number of channels of the feature map.

$${\overline{T}}_t=f_{LN}\left(f_{MSA}\right(t_p)+t_p)$$

(7)

$$T_t=f_{LN}\left(f_{FFN}\right(T_t^{\text{'}})+T_t^{\text{'}})$$

(8)

where $f_{LN}(·)$, $f_{MSA}(·)$ and $f_{FFN}(·)$ refer to functions of layer norm, multi-head self-attention (MSA) and feedforward network (FFN) respectively. MSA performs global correlation between textual semantic elements and outputs contextually enhanced textual features $T_t$.

To achieve depth alignment between text features $T_t$ and image features, a two-level multi-head cross-attention(MCA) is used. Where text feature $T_t$ and single character feature $X_S$ are used as first-level MCA inputs, with $T_t$ as the query, $X_S$ as the key and value, and let each character find the image feature that corresponds to it. The input tensors $X_S$ and $T_t$ are first split into n sub-tensors in the channel dimension.

$$Q_i=T_t{W}_i^Q,K_i=X_S{W}_i^K,V_i=X_S{W}_i^V, i=\mathrm{1,2},...,n$$

(9)

where $W_i^Q$, $W_i^K$, $W_i^V$ are the linear mapping matrices corresponding to the ith attention head respectively.

$$P_1^i=softmax\left({\displaystyle \frac{Q_i{K_i}^T}{\sqrt{d_k}}}\right)V_i$$

(10)

In this equation, $P_1^i$ is the calculated attention per head and $d_k$ is the length of $K_i$for scaling the attention. We process the results $P_1^i$ with a channel-wise concatenation operation $f_{con}$(·) and a linear projection function $f_w$, described as:

$${\overline{P}}_1=f_w\circ f_{con}(P_1^1,P_1^2,...,P_1^n)$$

(11)

The output ${\overline{P}}_1$ is passed to a FFN for feature refinement, obtaining a high-level prior $P_1$ without background interference.

In addition, the prior $P_1$ contains character image block $X_S$ as a key, text features $T_t$ as a value, and the size is consistent. It can be directly input into the secondary MCA to avoid dimensional inconsistencies. We use $P_1$ to connect semantic information and global visual features. We Flatten the shallow features $X_I$ as query. Unlike the $X_S$ in first-level MCA the $X_I$ are not flattened according to single characters, and the background and foreground information is not clear in terms of primary and secondary. However, $X_I$ contains global features, which effectively improves the visual quality problem caused by missing background. The $P_1$ as a key and $T_t$ as a value.

$${\overline{P}}_2=f_{LN}\left(f_{MCA}\right(X_I,P_1,t_p)+X_I)$$

(12)

$$P_2=f_{LN}\left(f_{FFN}\right({\overline{P}}_2)+{\overline{P}}_2)$$

(13)

Also each pixel point in $X_I$ can be queried by $P_1$ to the text feature corresponding to it. Obtaining a high-level prior $P_2$ for valid semantic mappings.

Since the cropped image is a sequence of image block features arranged by character, cluttered background interference is avoided when using transformer aggregation. And when calculating the correlation it reduces the interference of the neighbouring characters on the recovery of the current character. Global features $P_2$ can more accurately assign semantic information to the spatial domain via $P_1$.

3.3. Reconstructed Module

The reconstruction module in this study consists of a Sequential Residual Block (SRB), a Colour Correction Block (CCB), and an upsampling layer. $P_2+X_I$ sent to the SRB for SR reconstruction. And each time the resulting features are again sent to the SRB for element-by-element summation with $X_I$ to ultimately obtain the SR reconstruction features $X_{SR}^C$. However, the SR image restored through upsampling the reconstructed features $X_{SR}^C$ using the Pixel-Shuffle layer may suffer from color drifting in either the text part or the background part. In view of this, this study considers adaptive normalization to adjust the feature representation of the image to match the statistical properties of the target domain. So that the model can generate images that are consistent with the target style.

In detail, the mean and variance of the colour values of the SR image and LR image were first calculated using the $f_{SM}(·)$ function as the target criterion:

$${\sigma }_{X_I}^C,{\mu }_{X_I}^C,{\sigma }_{X_{SR}}^C,{\mu }_{X_{SR}}^C=f_{SM}(X_{SR}^C,X_I^C)$$

(14)

where c denotes the RGB channel, ${\mu }_{X_{SR}}^C$,${\mu }_{X_I}^C$ and ${\sigma }_{X_{SR}}^C$,${\sigma }_{X_I}^C$ denote the mean and variance for $X_{SR}^C$ and $X_I$. Secondly the $X_{SR}^C$ are normalized to improve the model accuracy:

$$X_{SR}^N={\displaystyle \frac{X_{SR}^C-{\mu }_{X_{SR}}^C}{{\sigma }_{X_{SR}}^C}}$$

(15)

Subsequently applying ${\mu }_{X_I}^C$ and ${\sigma }_{X_I}^C$ to the normalized feature $X_{SR}^N$, the SR image is adjusted so that its colour value mean and variance are consistent with that of the LR image to obtain the colour corrected reconstructed feature ${\overline{X}}_{SR}$.

$${\overline{X}}_{SR}=X_{SR}^N·{\sigma }_{X_I}^C+{\mu }_{X_I}^C$$

(16)

The reconstruction process in this study consists of five SRB and CCB modules connected in series, and finally an upsampling layer to obtain the reconstructed high-resolution image features $X_{SR}$.

3.4. Loss Function

In this study the total loss function consists of pixel loss $L_{pix}$ and text recognition prior loss $L_{tp}$.

$$L_{pix}={‖X_{SR}-X_{HR}‖}_2$$

(17)

The $L_{pix}$ loss generates high quality images from LR images by constraining the $L_2$ paradigm of super-resolution images and high-resolution images.

$$L_{tp}=\beta {‖A_{SR}-A_{HR}‖}_1+\gamma KL(t_p,t_{HR})$$

(18)

In Equation (18) $\beta$ and $\gamma$ denote hyperparameters, A denotes the attention map obtained using the text recognizer, and $t_p$ and $t_{HR}$ denote the text probability sequences of SR and HR images obtained by the text recognizer.$L_{tp}$ denote the text recognition branch is fine-tuned by constraining the $L_1$ paradigm and the KL scatter of the text prior recognized from the low-resolution image and the real image. The total loss function $\mathrm{L}$ is expressed as follows.

$$L=L_{pix}+\alpha L_{tp}$$

(19)

In Equation (19), $\alpha$ is the equilibrium parameter. During the training process, the loss function is used as the optimization objective, and the error signal is then passed to the proposed network through the back-propagation iterations to guide the updating of all the modules parameters.

4. Experimental Results and Discussion

In this paper, the DIFE is designed to dynamically adjust the multi-scale feature weights according to the different structures, shapes, and distributions of the input images in order to improve the information loss caused by single convolution. The DBFA as the main architecture of the model achieves deep interaction between global and local feature semantics to vision based on cross-attention. In addition the CCB adaptively adjusts the colour error caused by background interference using the input colour channel as a target criterion. The effectiveness of the components proposed in this study verified by ablation experiments in Section 4.2. Section 4.3 verifies the excellent performance of the model by performing comparative experiments on the TextZoom and the generalisation and robustness of this study is demonstrated by performing comparative experiments on the STR dataset.

4.1. Dataset and Experimental Details

TextZoom: The STISR dataset is derived from two state-of-the-art SISR datasets, RealSR and SRRAW, which are captured by a multifocal camera in the field, and are more realistic and challenging than the synthetic data. 17,367 pairs of LR-HR training sets and 4,373 pairs of test sets are included in TextZoom, with text annotations, border types, and raw focal lengths information. The smaller the focal length the blurrier the image is at the same height. The test set is divided into three subsets: 1619 samples for the easy subset, 1411 samples for the medium subset, and 1343 samples for the hard subset.

STR dataset: in order to verify the effectiveness of the method on different distribution datasets, we use five English STR datasets, namely, ICDAR2013 [36], ICDAT2015 [37], SVT [38], SVTP [39], and CUTE80 [40]. ICDAR2013 contains 1015 test samples, and ICDAR2015 contains 2,077 samples, and the text in the images may appear in a variety of scenes, with problems of distortion, occlusion, and uniform illumination. SVT contains 350 test samples, with large scale changes and complex backgrounds, the text may be bent and distorted, and the lighting conditions are varied, which makes the recognition more difficult. SVTP contains 645 test samples, although it is a synthetic data, the text image is of high quality, it lacks the complexity of the real scene. CUTE80 contains 288 text images with characters arranged along curved paths, forming curved text lines, with large image resolution, high quality and no LR images. In pre-processing, the images with resolution smaller than $16\times 64$ are selected and degraded with Real-ESRGAN [41] to test the robustness of the model.

The model is implemented using PyTorch framework and all the experiments are performed on a single RTX4090 GPU. The number of MSA in the SCBD is set to 4 and SRB is set to 5. The model with input batch size of 64, image width of 64 and height of 16 is optimized using Adam [42] and the number of training rounds is set to 500 and the learning rate is set to 0.001, which yields an output with a width of 128 and a height of 32. In this study, three text recognizer, ASTER, CRNN and MORAN, are used to assess the recognition accuracy. In order to assess the quality of image reconstruction, we use Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) [43] as reference metrics.

4.2. Ablation Experiment

In this study, the effectiveness of the DIFE, DBFA, and CCB is validated on the TextZoom using CRNN as a text recognizer.

4.2.1. The Role of Dual Branch Feature Aggregation

The DBFA is designed to accurately interpret the semantic information $t_p$ the corresponding positions in the image features $X_I$ and align between the textual a priori and the image features. Which is compared with other textual a priori interpreters in this study: firstly, the $t_p$ is fused with the $X_I$ using the inverse convolution block [9], secondly, the ${\mathrm{t}}_p$ and ${\mathrm{X}}_I$ are aligned using the SFT layer, and the SFT layer [44] outputs parameter pairs based on some textual a priori conditions, which affect the outputs by applying an affine transformation spatially to each intermediate feature map in the SR network to adaptively fuse the textual features. Finally, TPI is used to compute the correlation between textual prior and image features to guide the final SR text recovery. In order to verify the effectiveness of the DBFA, the local text feature aggregation LTFA is separately experimented as the text a priori interpreter.

Table 1 shows that the DBFA module proposed in this paper obtains the highest recognition accuracy and average accuracy in Easy, Medium, and Hard subsets, and the PSNR and SSIM indicators are also optimal, showing excellent SR performance. $f_D$ (50.6%) and $f_{SFT}$ (49.2%), although using text a priori to guide the reconstruction, they are not accurately assigned to the corresponding positions in the image feature space, resulting in the underutilisation of the a priori information, and TPI (52.8%), although aligning the image features with the text region well, the background foreground and foreground are not clear in terms of the primary and secondary, and the attention mechanism is not sufficiently allocated to the text part of the weights. Using only local textual feature integration (53.1%) ignores the background influence and reduces the visual quality. In contrast, DBFA focuses on both the foreground, accurately aligns the text prior and image features, and takes into account the background, improving the recognition accuracy to 53.6%, PSNR to 21.84, and SSIM to 0.7997, validating it as an effective super-resolution reconstruction method. The visual comparison in Figure 6 shows that the text image obtained by this paper's method has the best visual quality.

4.2.2. Dynamic Inception Feature Extraction

In this study, the DIFE is mainly used for multi-scale feature extraction[47]. Small convolutional kernels (e.g., 1x1, 3x3) focus on the local area, capture fine details such as edges and textures, and are suitable for extracting low-level features. Medium convolutional kernel (e.g. 5x5, 7x7) has a moderate sense field, captures both local and partial global information, and is suitable for extracting medium-level features such as part of an object's shape or silhouette. Large convolutional kernels (e.g., 9x9) have a large receptive field, cover a larger image area, and are suitable for extracting high-level features such as the overall shape of an object or the context of a scene. The dynamic weighting strategy adaptively adjusts the proportion of multi-scale feature information extracted from different convolutional kernels according to different input distributions. This experiment analyses the relationship between different sizes of convolutional kernels and whether or not to add the dynamic weighting strategy in feature extraction and recognition accuracy, and the results are shown in Table 2.

The combination of multi-scale convolution kernels proves to be advantageous by significantly improving the results compared to single convolution layer feature extraction. Small convolution kernels ignore contextual information to reduce accuracy, while large convolution kernels do not capture enough details. Experiments show that DIFE achieves the best recognition accuracy (53.6%) at the eighth set of convolution kernel sizes. The visualization in Figure 7 shows that the module is highlighted in the character region with the best retention of global and detailed features.

4.2.3. Validity of the CCB Module

To verify the effectiveness of the CCB module in colour correction, five models, TPGSR, TATT, C3-STISR, MNTSR, and SCE-STISR, are selected to compare the results with and without the CCB module. Table 3 shows that all models improve in text recognition accuracy or SR quality, proving that the colour correction module improves image reconstruction quality.

4.2.4. Effectiveness of Different Components

As shown in Table 4, this experiment investigated the effectiveness of introducing different modules, from which it can be seen that the results are optimal when all three modules are added at the same time.

4.3. Comparison with State-of-the-Art Results

4.3.1. TextZoom Quantitative Research

To verify the model validity, SCE-STISR was compared with SISR methods (SRCNN, SRResNet, EDSR, RCAN, CARN, HAN) and STISR models (TSRN, TBSRN, PCAN, TPGSR, Text Gestalt, TATT, C3-STISR, MNTSR, PerMR) were compared.

The recognition accuracies of the experiments based on CRNN, MORAN, and ASTER recognizer on the TextZoom test set are shown in Table 5. Since the SISR method is a generalised SR, it does not take into account the unique character structure of text images, and the textual information contained in the images. The results show that the recognition accuracy of SISR methods is generally low. Most of the other STISR methods use a single convolution layer, which is unable to take into account both character details and textual context, and treats foreground and background equally. with our method achieving the highest accuracy and sub-optimal average accuracy on the medium and difficult subsets; sub-optimal average accuracy on MORAN; and optimal simple subset and optimal average accuracy on ASTER. Compared with the STISR baseline TATT, we improve the average recognition accuracy on CRNN, MORAN, and ASTER by 1%, 1.4%, and 0.9%, respectively, demonstrating the effectiveness of the method.

The results of SSIM and SPNR, which are common evaluation metrics for super-resolution, are shown in Table 6. Our method employs a CCB to constrain the super-resolution reconstructed image to be infinitely close to the low-resolution image in the colour channel. So that the output image maintains the same statistical characteristics as the input, ensuring consistency with input perception, to improve the image visual quality, and structural similarity. While previous methods only consider the readability of the text part of the image, ignoring the effect of the image as a whole. Foreground and background interactions can also lead to colour drift, low reconstructed visual quality and weak structural similarity. The accuracy of the downstream text image recognition task depends on the image quality, and the PSNR of the model achieves the optimal results in the simple subset (24.99) and difficult subset (20.78) as well as the average PSNR (21.84), and performs the best in the medium subset (0.6955) and the difficult subset (0.7859) and the average (0.7951) SSIM metrics, which confirms its Validity.

4.3.2. TextZoom Qualitative Research

As shown in Figure 8, in order to further validate the effectiveness of the model, the visualization results after image reconstruction with different text lengths, backgrounds and colors and the text recognition results are shown.

Observations show that all methods outperform the bicubic interpolation method, but there is a significant difference in visual quality between the SISR and STISR. In this paper, the text-specific model SCE-STISR has the best visualisation results closest to HR images. As shown in the first and second columns of the figure, although most methods can accurately recognise characters, problems such as blurring of character structure and bending of strokes may occur when using text for super-resolution reconstruction. On the other hand, SCE-STISR can accurately assign semantic information to the corresponding image features, reconstruct the image correctly, and obtain satisfactory visual effects. In addition, the model performs well in dealing with the dense connection of characters and the background influence on foreground restoration, such as ‘recycled’ and ‘artificial’, the reconstruction results are clear and closest to the high-resolution image, while the other methods show the phenomenon of character mis-construction. In previous methods, the colour of characters or background is easily inconsistent with the colour of LR and HR images, resulting in large errors, such as ‘bucket’ and ‘caramel’, while this method effectively corrects the colour drift and significantly improves the fidelity of images.

4.3.3. Generalisation Capabilities on Text Recognition Datasets

In order to verify the generalisation ability of the model on other datasets, it is tested on five STR datasets, ICDAR2013, ICDAR2015, SVT, SVTP, and CUTE80, using the parameters trained on the TextZoom. These datasets are mostly derived from the real world and contain text of different lengths and complex background images. LR images are selected to form the test set, and due to the high quality of most of the images, Real-ESRGAN second-order degradation method is used to degrade the quality. As shown in Table 7, SCE-STISR can align the text prior and images more accurately, provide precise guidance, achieve better results, and prove the robustness of the model.

4.3.4. Discussions

Although the SCE-STISR network proposed in this paper can accurately reconstruct English characters and numbers, However, as shown in Figure 9, when the image does not contain characters in the text label reference character set, the recognizer may misrecognize it and cause reconstruction chaos. Second, the introduction of DBFA increases computational complexity and reduces inference speed. Future work will focus on designing networks that do not require textual prior guidance and exploring how to balance reconstruction quality and inference speed.