Deep Residual-Recurrent Networks with CTC for Unconstrained Urdu Nastaliq Handwritten Text Recognition: A Performance Gap Analysis

1. Introduction

Urdu is the national language of Pakistan [1] and a lingua franca across wide swaths of South Asia, with a native and second-language speaker base estimated at over 300 million plus in the whole world. [2] Its written form, the Nastaliq calligraphic script carries centuries of literary, religious, and administrative documentation. Enormous archives of handwritten Urdu material exist in libraries, government repositories, and private collections, yet remain effectively inaccessible to computational search, large-scale digitization, and natural language processing pipelines. The bottleneck is recognition: our previous work [3] concentrated on isolated character classification, a problem addressed by deep CNNs quite well, this chapter takes up a more difficult problem of recognizing character sequences within handwritten sentences without relying on character segmentation. With this shift, we change not only the problem from single classification to sequence but also the architecture from one of convolutional classifiers to that of convolutional recurrent sequence models.

In handwritten Nastaliq scripts, stroke shapes vary with writers, pens, and writing speed,while multiple joined characters vary in their forms as per their sequence in a word. Also signs and symbols are poorly positioned, and baseline positions change vertically. Production Urdu HTR can form the basis of digitizing legal archives, historic manuscripts and postal documents. Urdu corpora built via automated digitization can later be used for NLP research, including development of sentiment analysis tools, chatbots, handwriting style transfer and transliteration systems.

The Convolutional Recurrent Neural Network (CRNN) framework [4] along with Connectionist Temporal Classification (CTC) loss [5] helps in solving these situations in OCR. A CNN is used as a feature extractor, from which features are stacked to become a sequence which is latter given as an input to an RNN to determine the sequence of the most likely characters. The convolutional neural networks approach was improved with the Shi et al. [5] work on Convolutional Recurrent neural Network (CRNN), combining convolutional sequence extraction with a bidirectional recurrent neural network and the Connectionist Temporal Classification (CTC) loss objective into one training pipeline. CTC avoids an issue that hinders HTR from scripts where boundaries are ambiguous, such as Nastaliq (a South Asian script extensively used in Persian, Urdu, and Kashmiri literature).

The script’s inherent structure, which is sometimes visually linked with other characters, are often not separated, creating a set of characters that makes a string of characters indistinguishable from other strings without information provided by character segmentation or contextual clues. The CTC system allows learning from data labelled at the character line, than at word or character level. In this system, after image columns are converted to sequential 47 feature representations, an RNN network makes a prediction based on the input sequence. As with CRNN systems for other languages, CTC handles this ambiguity since the model can potentially generate long sequences with blanks at specific time steps which can be filtered later during inference.

We also add bidirectional LSTMs to CRNN to simplify the complex and constantly changing shapes of Nastaliq letters to make the script more uniform, since LSTMs are capable of propagating context across long time periods. Although early work on Nastaliq HTR was published as far back as 2013, utilizing techniques including multi-dimensional LSTMs [6] or a hybrid convolutional-recurrent approach [7] on handwritten datasets or on printed text, they were far less deep compared with state-of-the-art systems published today and generally started their training from a random initialization with a smaller and simpler dataset, never reaching accuracy figures close to those for Latin script handwritten digits or words, etc.

Another path to image representations in computer vision is deep residual networks [8], along with their ImageNet-trained counterparts. One can work with these deep architectures as feature extractors, transfer their backbones to HTR, but the resulting structure clashes. A typical deep network that classifies images down samples by a factor of 2 at the beginning of stages 2-4, there by halving the width of the feature maps from the input width of a 256px image down to 8 columns. It down samples every step for images up to hundreds of pixels wide with width as narrow as 3x at the end of its pipeline. Deep neural networks are built to recognize specific patterns (like edges, shapes, or objects) and keep recognizing them no matter where they appear in an image. To do this efficiently, the network compresses the data as it gets deeper. Unfortunately, the required length of the encoded sequence is T ≥ S for CTC. Using a 256-pixel-wide line of text yields features as thin as 8 pixels in width, for practical purposes, no Nastaliq line can be accurately transcribed with these number of columns. Despite the generalization improvement from transfer learning with these massive models and ImageNet data, the lack of spatial depth makes the use of ImageNet features infeasible with CTC in Urdu HTR systems. On the other hand, attention and Transformer networks have adopted and surpassed these representations, achieving CER rates in the single digits, like with the ET-Network [9] reporting 5.27% CER on the NUST-UHWR dataset. Unfortunately, this level of performance typically comes at a high cost. For researchers, there are large, annotated datasets and augmentation procedures, as well as computationally expensive deep learning models and infrastructure that are not accessible to all. We build a model that is situated between these two approaches. The first contribution is an experiment based on replacing the (2, 2) down-sampling strides with unit strides in residual stages 2, 3 and 4 of the ResNet34 network. These changes maintain the width of the features map T = W/4 throughout the depths while ensuring the CTC constraint, so that the minimum width is 4 x (8+6-1) = 52 pixels. The resulting model is trained and evaluated using a small subset of 7,300 lines of Urdu handwriting, achieving a baseline Character Error Rate (CER) of 32.38%. This approach is superior to prior CTC baselines, as it does not rely on generated synthetic data, a complex non-linear augmentation pipeline or autoregressive sequence decoding. Second, a gap analysis is performed along three directions: decoding strategy, the scale of data available and the effectiveness of augmentation. We analyse why 32.38% remains an upper bound to current Urdu HTR standards and not a lower one, outlining the direction to reduce the current distance of 27.11% down to the reported results from state-of-the-art models.

2. Background

2.1. From Individual Characters to Lines of Text

It is one of the earliest problems to occur in pattern recognition, dating back to the early 1960s. Early character recognition systems usually imposed harsh limitations: they only worked for handwritten letters separated out from the background (e. g., printed characters or letters written by skilled calligraphers), required small vocabularies of restricted length and demanded writers to be precise. Going from simple letter identification to recognition of handwritten cursive lines was a challenging task and took about 30 years to solve. Plamondon and Srihari’s article [10] gives a broad survey of this evolution of the problem, they distinguish two major paradigms of handwriting recognition: online handwriting recognition which collects the sequence of points (x, y) drawn by the pen in time-order while the writer is writing, versus offline handwriting recognition where the system works on static image. Offline recognition is the harder task. Since it lacks a time signal, only a pattern formed by light reflected from paper exists to identify the characters and words on the document. Offline handwriting recognition dominated throughout the 1990s and into the 2000s through HMM-based methods [11]. They assume text to be an unknown sequence of states that produces observable emission features from vertical slices in the text. By means of Viterbi’s algorithm, these systems decode the most likely sequence of states and achieve good recognition rates on Latin-script documents with controlled image acquisition and feature extraction using gradient histograms, projection profiles or run length distributions. [12] However, the conditional independence of observations in different slices of vertical text strips causes these HMMs to lose dependence information over distance. Thus, they disregard the long-range correlations between strokes in the same handwritten letter, which are important for visual character recognition.

2.2. The Nastaliq Script

Unlike Latin scripts Arabic-based scripts provide challenges for sequential recognizers and Urdu-Nastaliq is the preferred style in places where Urdu is written on daily basis, and is perhaps the most challenging script visually as it contains more characters than Arabic. Its difficulties are magnified due to three properties. Firstly, Nastaliq is a tied script. The elements of which are called huroof, and they do not typically appear in isolation but in a combination nuqtas and tied clusters. These forms often combine characters into a unit that is not a mere addition but a merging into a completely new shape which can correspond to anything from two to eight characters. Segmenting these complex forms before trying to recognize the text would not only be fragile but lossy as well since segmentation destroys exactly the hints required for robust recognition.

Second, the script is written from right-to-left. Whereas Latin script tends to be confined into more of a horizontal plane, Nastaliq lines contain secondary strokes and marks (e. g., zabar, zer, pesh, jazm) sit above and below the primary characters. It is worth remembering that these strokes and marks are not merely cosmetic, they mark vowels of length 1 or 2 and they carry semantics and can be destroyed by improper straightening or segmentation.

Third, the characters are context dependent. Take, for example, the letter kaf (ک) depending on its placement within a word and the specific neighboring characters, it has four different shapes [13], none of which can be resolved without knowledge of the environment. Context here is not a peripheral detail but is the script’s organizing principle. The interaction of all three traits creates an environment where it is not desirable but essential to perform segmentation-free recognition by directly modelling text with sequence-to-sequence architectures and, therefore any system segmenting text explicitly as an intermediate step will lose the contextual clues that make it possible to classify individual character forms.

2.3. CRNN and CTC

Neither CRNN nor CTC were individually revolutionary, instead, deep learning was introduced into HTR through the application of powerful general tools. The Hochreiter and Schmidhuber LSTM [14] had already been developed which showed that RNNs could backpropagate gradients along sequences of over 1000 steps, avoiding vanishing gradients and making the long range possible. In 2006, Graves et al. [4] had shown how to use CTC loss to remove the 50 need for any alignment between frame sequences of image sequence inputs and character-level targets. CTC works by integrating over all possible ways to align the target character sequence to the image frame sequence, creating an end-to-end trained system that outputs probability distributions over characters that the RNN system outputs. The combination of these three methods was coined CRNN in Shi et al. [5]. Their architecture treats the image as a line strip, using convolutional layers for local feature extraction on successive columns (which produce output features in a sequence of one column feature vector per strip slice). BiLSTMs then extract long range dependencies on this sequence which are finally decoded by CTC loss into a final text string. No character bounding boxes or masks were required by their CRNN training procedure, alignment between specific character regions and output characters was learned solely based on the full-text line level annotations.

The method was effective for Arabic-family scripts. Naz et al. [6] trained MLSTM networks on printed Urdu Nastaliq, attaining 98% accuracy by allowing a recurrence that could travel in the horizontal as well as diagonal directions, demonstrating that the two-dimensional structure could be learned. The same team again [7] then employed feature maps that are first extracted via convolutional layers, followed by recurrent MDLSTM processing, they reported improvement on a partially restricted handwritten version of Nastaliq.

2.4. Temporal Resolution Bottleneck in Deep Convolutional Backbones

CRNN rapidly gained popularity, but practitioners substituted shallow, pretrained convolutional neural networks as ResNets [8], DenseNets, and EfficientNets. But for the hand designed narrow convolutional stack they encountered problems by gradually reducing the input width. This type of network created translational changes in its outputs, progressed convolutions or pools consistently reduce feature map dimensions by half while paralleling increasing the receptive field. This was fine until classification was the concern, but the final features should not depend on the specific position of the target, e.g. the specific position of the left eye in the center of an image. However, CTC losses demand exactly the opposite way i.e., minimum output width or height T, is bounded below by max length S of any label sequence such that T >= S.

A ResNet34 with default strides would halve width of a 128px-wide Nastaliq image at every step. After its third residual stage, at a point that still requires a classification, the feature maps will be 4 pixels wide. A text line may contain fifteen characters on average while the longest line may reach twenty characters, if T=4 and S=15, then 4<15, and CTC probability would be zero to all possible paths and the loss would grow without limit. This issue with 51 inadequate temporal resolution appears across multiple script recognition research and was identified as an important problem in adapting image classification backbones to OCR work [15,16]. The standard practice now consists of replacing every (2,2) stride at deep residual stages with a (1,1) stride, changing layers that are responsible for down sampling at expense of width to layers that preserve width while building feature complexity. That is a trade-off: The increase in size of feature maps after such a modification while the overall length S still holds, could result in a significant increase in memory use. However, this is crucial for the solution used in this chapter, since it makes these ImageNet transfers useful.

2.5. Attention Mechanisms and Transformer-Based HTR

With advances in CRNN-CTC, another line of research was quietly emerging which would soon surpass them. A encoder/decoder architecture with attention [17] offers yet an alternative to formulate the task other than CTC, instead of assuming that each prediction output step is conditionally independent given the encoding representation of each input symbol, the decoder attends to certain parts of the feature map that are relevant for producing a particular next character, as can be seen with many cursive characters (i.e. Letters have drastically different forms depending on the preceding context). The next character is determined, at each step, on the selected region of the encoding feature map and from the context-based language information derived from the previously predicted sequence.

The transformer architecture [18] took the attention mechanism one step further and removed recurrence, replacing LSTM recurrent connections entirely with multi-head self-attention mechanisms, facilitating parallelization over mini-batches and thus permitting massive-scale parallel training. Subsequently, researchers [19] found that even convolutional encoders can be replaced by a Transformer encoder (treating image chunks as sequences of tokens) and a wave of research applied these findings to the HTR field, with, e. g., Anjum and Khan [17] introduce a large, annotated dataset of handwritten Urdu sentences and presented a methodology for the recognition of offline handwritten Urdu text lines. A deep learning-based encoder/decoder framework with attention mechanism to handle two-dimensional text structure. Their proposed model claimed improvement on BLSTM based state-of-the-art by a factor of 2 in terms of character level accuracy and by a factor of 37 in terms of word level accuracy on their dataset.

One of the leading works recently, the ET-Network [9], introduces self-attention to an Efficient Net encoder combined with an autoregressive Transformer-language model decoder. This model achieved CER scores of 5.27% and WER of 19.09% on the NUST-UHWR, hence setting the performance benchmark targeted by this paper. The ET Network has improved over the 52 CTC by four orders of magnitudes firstly an enriched feature encoding pipeline using ETNet, secondly an autoregressive decoder with language modelling abilities, thirdly training data which is 3-4 times larger than NUST-UHWR and applying many methods of elastic data augmentation to mimic the variability in freeform ink of free-style pen strokes in a more accurate way.

2.6. Data Scarcity and Augmentation

One key aspect is that data quantity is more important than architecture for visually challenging scripts. Bluche and Messina [16] proposed a new neural network architecture for state-of-the- art handwriting recognition, alternative to multi-dimensional long short-term memory (MD- LSTM) recurrent neural networks. The model is based on a convolutional encoder of the input images and a bidirectional LSTM decoder predicting character sequences. In this paradigm, they aimed at producing generic, multilingual and reusable features with the convolutional encoder, leveraging more data for transfer learning. The architecture was also motivated by the need for a fast training on GPUs, and the requirement of a fast decoding on CPUs. The main contribution of their paper lies in the convolutional gates in the encoder, enabling hierarchical context-sensitive feature extraction. Their experiments on a large benchmark including seven languages show a consistent and significant improvement of the proposed approach over their previous production systems. They also reported state-of-the-art results online and paragraph level recognition on the IAM [20] and Rimes [21] databases.

2.7. Contribution

High-performance Urdu HTR is, in principle, well-solved on several popular benchmarks where a massive dataset of handwritten text is available along with specialized augmentations to create synthetic data to meet the requirements of autoregressive Transformer-based systems, for example. However, a fully reproducible and lightweight method that allows good transfer learning results has not been thoroughly documented. Urdu Nastaliq handwriting using the CTC objective function has primarily consisted of shallow custom CNN architectures without ImageNet preprocessing [6,7] or does not provide a granular component-level explanation of why it is still below standard Transformer systems.

In this work, we fulfil both roles. First, we take the common ResNet34 backbone and modify only stride parameters to comply with the length requirement posed by the CTC mechanism and show how the framework would look and operate when trained on just a fraction of the Urdu data and thus provide a baseline which is both reproducibly implemented and theoretically valid. Second, we proceed to show the source of the existing performance degradation by performing a systematic analysis along three dimensions: decoding approach, training data availability and the inclusion/modification of training augmentations. Together, these analyses form an unprecedentedly granular view of the remaining gap which the community must close to reach the state-of-the-art performance on this task.

3. Dataset

Ahmed et al [22] presented comprehensive Urdu handwritten offline database name Urdu- Nasta’liq handwritten dataset (UNHD). UNHD database covers all Urdu characters and ligatures with different variations. Their dataset can be used for handwritten character recognition as well as writer identification. The vocabulary of UNHD database spans digits, numbers, and part of speech noun, verb, pronoun, etc., but in Urdu Nasta’liq font. They have acquired samples from school and college students, office going individuals to ensure that we must have all variability of handwritten samples from all individuals associated to any field and from every age. The data collected from 500 writers (both male and female) to broaden the collected text from 48 unique text lines to 700 unique text lines including Urdu numerals and Urdu constraint handwritten samples with have more than 6000 Urdu handwritten text lines.

Few examples from the dataset are given in Figure 1 showing the raw image and the ground truth text.

Figure 1. Dataset examples of raw images with the ground truth.

4. Proposed Methodology

4.1. System Architecture Overview

The proposed system is a three-stage pipeline operating end-to-end on a single text-line image. A modified deep convolutional backbone extracts a dense sequence of local visual features. A bidirectional recurrent network then models the temporal dependencies across this sequence. Finally, a CTC transcription layer maps the recurrent output to a probability distribution over character sequences without requiring any alignment supervision. Figure 2 illustrates the complete architecture.

Formally, let an input image be represented as $\mathrm{X}\in {\mathbb{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}$ where H, W, and C denote height, width, and number of input channels (C = 1 for grayscale, C = 3 for RGB). The convolutional encoder φ maps X to a feature sequence: $\mathrm{F}=\mathsf{\phi }\left(\mathrm{X}\right)\in {\mathbb{R}}^{\mathrm{T}\times \mathrm{d}}$, where T is the number of time steps in the encoded sequence and d is the feature dimensionality at each step. The recurrent module ψ transforms F into a context-enriched sequence: $\mathrm{H}=\mathsf{\psi }\left(\mathrm{F}\right)\in {\mathrm{R}}^{\mathrm{T}\times 2\mathrm{h}}$, where h is the number of hidden units in each directional LSTM and the factor of 2 arises from concatenating forward and backward states. A fully connected projection layer ${\mathrm{W}}_{\mathrm{o}\mathrm{u}\mathrm{t}}\in {\mathrm{R}}^{2\mathrm{h}\times \left|\mathrm{A}\right|+1}$ maps each time step of H to log-probabilities over the character alphabet A augmented with the CTC blank token ε: $\mathrm{Y}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\mathrm{W}}_{\mathrm{o}\mathrm{u}\mathrm{t}}\mathrm{H}+{\mathrm{b}}_{\mathrm{o}\mathrm{u}\mathrm{t}}\right)\in {\mathrm{R}}^{\mathrm{T}\times \left(\left|\mathrm{A}\right|+1\right)}$. The CTC objective is then applied to Y and the target label sequence $\mathrm{l}=\left({\mathrm{l}}^1,{\mathrm{l}}^2,\dots ,{\mathrm{l}}_{\mathrm{S}}\right)$ during training.

4.2. System Visual Feature Extraction: Modified ResNet34 Backbone

4.2.1. Residual Learning

The standard residual block, introduced by He et al. [8], wraps a stack of convolutional layers $F\left(·\right)$ with a shortcut (identity) connection: $y = F\left(x,\left\{W_i\right\}\right)+ x$, where x and y are the input and output feature vectors of the block, $F\left(x, \left\{W_i\right\}\right)$ represents the residual mapping to be learned, and the addition is element-wise. When the input and output dimensions differ as occurs at strided layers, the shortcut is replaced by a learned linear projection: $y = F\left(x,\left\{W_i\right\}\right)+ W_s x$, where $W_s$ is a 1×1 convolution that matches dimensions. Each residual function F consists of a sequence: BatchNorm → ReLU → Conv (3×3) → BatchNorm → ReLU → Conv (3×3). This formulation allows gradient to flow directly from any output layer back to any earlier layer through the identity path, mitigating the degradation problem in very deep networks.

ResNet34 stacks 16 such basic blocks across four residual stages (layer1 through layer4), producing progressively deeper feature representations. The network was pretrained on ImageNet-1K, whose scale (~1.28 million images, 1000 classes) provides a rich initialization for low-level texture, edge, and shape detectors directly applicable to handwriting images.

The core feature extractor is a ResNet34 backbone where stride parameters in `layer2`, `layer3`, and `layer4` are modified to (1,1). This ensures the output feature map has a sequence length 1/4th of the original image width, providing enough granularity for CTC alignment. A 2-layer BiLSTM models the character dependencies before a linear projection to the vocabulary space.

4.2.2. The CTC Length Constraint and the Stride Modification

The CTC loss requires that the encoded sequence length T satisfies: T ≥ S, where S is the number of characters in the target string. Violation renders the CTC probability for any correct decoding identically zero, causing the loss to be undefined. For a standard ResNet34 applied to an input of width W, the output feature map width after each stage is given in Table 1.

For W = 256 (a standard text-line image width after normalization), $T_{standard}= 8$.

A Nastaliq text line containing S = 15–25 characters, entirely typical for unconstrained writing that violates the constraint T < S with certainty. CTC training on such a backbone is impossible without modification.

The proposed fix replaces the spatial down sampling strides in layer2, layer3, and layer4 with unit strides as shown in Table 2.

The shortcut projections within those first blocks are likewise changed to stride (1,1). The resulting feature map dimensions under this modification become as in Table 3.

The encoder sequence length is thus T = W/4, which for W = 256 yields T = 64. This satisfies the CTC constraint for all realistically encountered Nastaliq line lengths while preserving the full representational depth of the 34-layer backbone.

Formally, the modified encoder applies a strided initial convolution (7×7, stride 2) followed by max-pooling (3×3, stride 2), reducing both spatial dimensions by a factor of 4 total. All subsequent residual stages preserve the horizontal dimension through unit-stride convolutions. The vertical dimension continues to collapse through the default spatial operations, ultimately being pooled away before the recurrent stage. The output feature sequence can be written as: $F = {\phi }_{\mathrm{m}\mathrm{o}\mathrm{d}}\left(X\right) \in {\mathbb{R}}^{T\times 512}$, where 512 is the number of output channels in layer4 of ResNet34. Each column $f_t\in R^{512}$ aggregates visual information from a receptive field of width approximately $W/T = 4$pixels, providing fine-grained spatial coverage suitable for dense Nastaliq character boundaries.

4.3. Temporal Sequence Modelling: Bidirectional LSTM

A single text-line image column does not carry sufficient context for unambiguous character identification. The visual identity of a Nastaliq glyph is entangled with its neighbors through ligature merging; the encoder feature $f_t$ at column t encodes only local information. The recurrent stage models the global sequence context.

4.3.1. LSTM Cell Equations

The LSTM cell [14], which resolves the vanishing gradient problem through explicit memory gating, is governed at time step t by: $i_t = \sigma \left(W_{xi} x_t + W_{hi} h\left\{t-1\right\}+ b_i\right)$, $f_t = \sigma \left(W_{xf} x_t + W_{hf} h\left\{t-1\right\}+ b_f\right)$, $g_t =\tanh \left(W_{xg} x_t + W_{hg} h\left\{t-1\right\}+ b_g\right)$, $o_t = \sigma \left(W_{xo} x_t + W_{ho} h\left\{t-1\right\}+ b_o\right)$, $c_t = ft\bullet c\left\{t-1\right\}+ i_t\bullet g_t$ and $h_t = o_t\bullet \tanh \left(c_t\right)$Where $x_t\in R^d$ is the input at time t, $h_t\in R^h$ is the hidden state, $c_t \in R^h$ is the cell (memory) state, σ denotes the sigmoid activation, ∙ denotes element-wise (Hadamard) product, and W{·} and b{·} are learned weight matrices and bias vectors. The input gate $i_t$ controls which new information enters the cell. The forget gate $f_t$ controls what is erased from the previous cell state. The output gate $o_t$ modulates how much of the cell state is exposed as the hidden output. The cell state $c_t$propagates through time via the forget gate, enabling the network to maintain gradient signal over long dependencies.

4.3.2. Bidirectional Extension

A unidirectional LSTM reading left-to-right encodes only the leftward context at each position. For a right-to-left script like Urdu, the reading direction is reversed, but the deeper problem is that a single-direction recurrence cannot represent bilateral context that is the character appearing after a given position is as diagnostic as the character before it, particularly for resolving allographic ambiguity.

The BiLSTM [23] addresses this by running two independent LSTM chains over the sequence, one in the forward direction and one in the reverse, then concatenating their hidden states: $\to h_t = LST{M}_{fwd}\left(xt,\to h\left\{t-1\right\}\right)$, $\leftarrow h_t = LST{M}_{bwd}\left(xt,\leftarrow h\left\{t+1\right\}\right)$ and $h_t = \left[\to h_t ; \leftarrow h_t\right]\in R^{2h}$. The concatenated state $h_t$ thus encodes context from the entire input sequence in both directions simultaneously. This is a strong structural prior for Nastaliq: the shape of a stroke at position t is co-determined by all prior and subsequent strokes in the ligature chain.

The proposed model stacks two such BiLSTM layers, each with h = 256 hidden units per direction, for a final hidden dimension of 512. Dropout is applied between the two BiLSTM layers during training to regularize the recurrent states.

4.4. Sequence Transcription: Connectionist Temporal Classification

4.4.1. The Alignment Problem

Given an encoder output sequence F of length T and a target character string $l = \left(l^1, \dots , l_S\right)$with S ≤ T, the fundamental challenge is that the ground-truth alignment between time steps and characters is unknown. For Nastaliq, where a single ligature may span many pixel columns, the visual “boundary” between one character token and the next has no canonical definition. CTC [4] resolves this by treating all possible alignments as latent variables and marginalizing over them during training.

4.4.2. The CTC Alphabet and Path Definition

Define the extended alphabet $Ā = A \cup \left\{\epsilon \right\}$, where A is the character vocabulary and ε is a special blank token. A CTC path $\pi \in {Ā}^T$ is a sequence of T labels, one per time step, drawn from Ā. The probability of a path π given the network output Y is: $p\left(\pi |X\right)={\prod }_{\left\{t=1\right\}}\left\{T\right\} y_{\left\{{\pi }_t\right\}}^{\left\{t\right\}}$, where $y_k^t$ is the probability of label k at time step t. The collapsing function B maps any path π to a label sequence by first removing consecutive repeated labels, then removing all blank tokens: $B\left(\pi \right)= remov{e}_{\epsilon }\left(deduplicate\left(\pi \right)\right)$. For example, B(ε, ب, ب, ε, ا, ε) = (ب, ا). The set of all paths that collapse to the target sequence l constitutes all valid alignments: $B^{-1}\left(l\right)= \left\{\pi \in {Ā}^T: B\left(\pi \right)= l\right\}$.

4.4.3. CTC Loss Computation

The CTC loss for a single sample is the negative log probability of the target string l under all valid alignments: $L_{CTC\left(l, X\right)} = -\mathrm{l}\mathrm{o}\mathrm{g}p(l | X) = -\mathrm{l}\mathrm{o}\mathrm{g}\sum _{\pi \in B^{-1}}\left(l\right)\bullet p(\pi | X)$. Naïve enumeration of $B^{-1}\left(l\right)$ is exponential in T. The efficient computation uses the forward-backward algorithm on the extended label sequence $l^{\text{'}}= \left(\epsilon , l^1, \epsilon , l^2, \dots , \epsilon , l_S, \epsilon \right)$ of length 2S + 1.

The total probability of the target is obtained by summing over the terminal states: $p\left(l \right|X)=a_T\left(2S+1\right)+a_T(2S)$. The gradient of the CTC loss with respect to the pre-SoftMax output (the logit) $z_k^t$ is computed from the combined forward-backward terms: ${\displaystyle \frac{\partial L_{CTC}}{\partial z_k^t }}= y_k^t - (1 / p(l | X)) \sum _{s : l_s^{\text{'}} = k}{\alpha }_t\left(s\right)·{\beta }_t\left(s\right)$. This gradient is backpropagated through the SoftMax, through the projection layer, through the BiLSTM, and through the convolutional backbone in a single end-to-end pass. No alignment, segmentation, or intermediate supervision is required at any stage.

4.4.4. Inference: Greedy CTC Decoding

At inference time, the simplest decoding strategy collapses the per-step maximum-probability prediction: ${\pi }^{*} = argma{x}_{\pi } p(\pi | X) = {\left(argma{x}_k{y}_k^t\right)}_{t=1}^T$. The predicted label sequence is then obtained by applying the collapsing function:$l̂ = B\left({\pi }^{*}\right) = B(argma{x}_k y_k^1, argma{x}_k y_k^2, \dots , argma{x}_k y_k^T)$. This greedy approach is O(T|A|), essentially free at inference. Its suboptimality relative to full CTC beam search [24] is bounded in practice when the BiLSTM hidden states are expressive, but it is notably degraded when the model assigns diffuse probability mass over competing characters, precisely the scenario caused by visual ambiguity in Nastaliq allographs. The absence of beam search with an external language model is therefore a deliberate scope boundary of this baseline: it isolates the visual recognition component from language model post-processing and ensures the reported CER reflects the model’s intrinsic visual discrimination capacity.

4.4.5. Training Configuration and Optimization

The full model was trained end-to-end from the ImageNet-pretrained ResNet34 weights with the following configuration:

Loss function: CTC loss applied to the projected BiLSTM output over a vocabulary of |A| Urdu characters plus the blank token.

Optimizer: Adam with initial learning rate η₀ = 1×10⁻³, β₁ = 0.9, β₂ = 0.999.

Learning rate schedule: Plateau-based reduction, the learning rate was multiplied by a factor of γ = 0.1 whenever the validation CTC loss failed to improve over a patience window of P = 10 epochs.

Input normalization: Text-line images were height-normalized to H = 64 pixels with width-proportional resizing, and pixel values were normalized to zero mean and unit variance channel-wise using ImageNet statistics.

Data augmentation: Affine transformations with random rotation within ±5°, horizontal and vertical shear within ±2°, and random scaling within ±10% were applied on-the-fly during training. No elastic (non-linear) augmentation was applied in this baseline configuration; its absence is one of the three primary contributors to the generalization gap characterized in Section 3.4.

Training duration: The model was trained for a maximum of 200 epochs, with early stopping when the validation CER failed to improve for 20 consecutive epochs. The minimum validation CER of 32.38% was reached at epoch 127, after which validation loss plateaued at 1.61 against a training loss of 0.51 which is a discrepancy of ≈ 1.10 nats that quantifies the magnitude of the overfitting induced by limited training data.

Hardware and implementation: All experiments were conducted in PyTorch. The modified ResNet34 backbone was initialized with torchvision.models.resnet34(pretrained=True), with stride parameters modified in-place before training commenced.

5. Results And Discussion

The experimental results are organized across five visualizations, each designed to communicate a distinct dimension of the model’s behavior. Figure 3 and Figure 4 document the learning dynamics; Figure 2 describes the system architecture; and Figure 5 decomposes the performance gap into its constituent axes. Together these figures build a cumulative argument: the proposed system is a sound engineering baseline, its ceiling is identifiable and explainable, and the path toward SOTA performance is tractable.

5.1. Comparative Performance

Table 4 spans three generations of Urdu HTR methodology, from statistical sequence modelling through early convolutional systems to the present deep residual baseline and current Transformer state of the art. Read top to bottom, the table narrates an arc of steady improvement, but that arc is not smooth. The transition from the proposed CTC baseline at 32.38% to the attention-based model at approximately 7.00% represents a discontinuous performance jump of more than 25 percentage points, not a gradual refinement. This non-linearity is the table’s most important observation. It reflects a qualitative architectural shift, the introduction of an autoregressive decoder with explicit language modelling capacity, rather than any incremental improvement in encoder quality or training scale. The proposed model improves on the shallow deep learning baseline by 3.12 percentage points using a comparable training corpus and a similar CTC decoding strategy, a gain attributable specifically to the depth and ImageNet pretraining of the modified ResNet34 encoder. That the encoder upgrade alone yields a measurable improvement, while still leaving a 27-point gap to attention-based systems, confirms that encoder quality is a necessary but insufficient condition for competitive Nastaliq recognition. The ET-Network’s 5.27% CER, representing the current state of the art, was achieved with approximately 2.7 times more training data and a substantially richer augmentation pipeline alongside its Transformer decoder; the independent contribution of each of these advantages is difficult to isolate from the published results and is addressed directly in the gap analysis of Section 5.3.

5.2. Learning Dynamics

Figure 3 and Figure 4 tell the same story from two perspectives. The loss curves in Figure 3 show the training CTC loss descending continuously throughout the run, reaching a floor of 0.51 at epoch 127 — evidence that the model has adequate capacity to fit the training partition. The validation loss, by contrast, stabilizes around 1.61 nats after approximately epoch 60 and declines no further, producing a generalization gap of 1.10 nats that persists to the end of training. This divergence is not a sign of model over-parameterization in the classical sense. Rather, it marks the point at which the model has effectively memorized the writer pool present in the training subset: further gradient steps refine its representation of those specific handwriting styles rather than the script’s underlying visual structure. The plateau-based learning rate scheduler registers this stagnation and triggers a tenfold reduction in learning rate, producing a small step-down improvement in CER visible around epoch 100 before early stopping is activated at epoch 127.

Figure 4 recasts the same trajectory as recognition performance, showing the validation CER descending from near-100% at initialization to its minimum of 32.38%. The descent is steep through the first 30 epochs, then progressively flattens as the model approaches its data-imposed ceiling. The three reference lines drawn at 35.50%, 7.00%, and 5.27% provide essential context: the model clears the shallow baseline comfortably but never approaches the attention or Transformer thresholds. This ceiling is not a training failure rather the model converged, and the learning rate schedule responded appropriately. It is a structural consequence of the three deficits quantified in Section 4.3.

5.3. Comprehensive Gap Analysis

To understand why the proposed model achieves 32.38% while SOTA models achieve <10%, we performed a gap analysis across three pillars: architecture, data, and compute.

Table 5 and Figure 5 together constitute the paper’s primary analytical contribution. Where Table 4 reports what the numbers are, Table 5 explains why they are what they are, disaggregating the 27.11 percentage-point gap between the proposed baseline and the ET-Network into four independently addressable components: decoding strategy, training data volume, augmentation sophistication, and feature.

The radar chart in Figure 5 renders this decomposition geometrically. The proposed model’s polygon nearly closes on the SOTA across the encoder quality axis is a direct consequence of ImageNet pretraining but collapses sharply inward on decoding strategy and data volume, which are the two axes carrying the most explanatory weight. The geometric asymmetry is deliberate: it makes immediately visible that the gap is not monolithic and cannot be closed by any single intervention. A reviewer who examines Figure 5 before reading the body text will apprehend the paper’s central argument at a glance, the encoder is competitive, the decoder is the bottleneck, and the data deficit is the second most critical constraint. This prioritization has direct implications for the future work agenda: targeting the decoding axis first, through shallow language model fusion or a hybrid CTC-attention architecture, is likely to yield the largest marginal return per unit of implementation effort and is accordingly addressed first in Section 7.

5.3.1. The Architectural and Decoding Gap

Our framework uses CTC with greedy decoding, which assumes frame-wise character independence. This places the entire burden of spelling and context on the BiLSTM’s hidden states. Conversely, autoregressive decoders in Transformer models act as internal language models. If the visual feature of a character is ambiguous, the decoder uses previous character history to “spell” the word correctly, a feature currently missing in our pure CTC setup.

5.3.2. The Dataset and Augmentation Gap

Modern SOTA models utilize datasets three to four times larger than our subset and rely heavily on Elastic Transformations. Linear affine shifts cannot replicate the natural variation in baseline slope and pen pressure that elastic warping provides. This “data starvation” explains the plateauing validation CER observed in Figure 4.

5.3.3. The Computational Gap

High-performance models typically require hundreds of additional epochs and specialized hardware for long-range attention calculations. Our baseline offers a more efficient convergence profile but with a lower predictive ceiling, establishing it as a lightweight alternative for resource-constrained environments.

6. Conclusion

This paper set out to answer a constrained but consequential question: how well can a deep, pretrained convolutional backbone perform on unconstrained Urdu Nastaliq handwriting recognition when coupled with a standard CTC transcription framework and trained on a limited data subset? Character Error Rate of 32.38% is not a number to celebrate in isolation. It is a number to interrogate.

The principal engineering contribution is the stride modification strategy applied to the ResNet34 backbone. Replacing the (2,2) strides in residual stages 2 through 4 with unit strides dissolves the fundamental incompatibility between deep down sampling architectures and the CTC length constraint (T ≥ S). The resulting encoder preserves a horizontal sequence length of T = W/4 across all residual depth, retaining the full representational capacity of an ImageNet-pretrained network while making it CTC-compatible. This is a technically straightforward modification, but its reproducibility and principled motivation are precisely what the literature has lacked for Urdu-family scripts. Prior CTC-based Nastaliq systems either used shallow custom convolutional stacks that foregoed transfer learning entirely or did not isolate this stride bottleneck as an explicit design variable.

The second contribution is the systematic gap analysis. By decomposing the 27.11 percentage-point distance between this baseline (32.38% CER) and the current Transformer state of the art (5.27% CER, ET-Network) into three independent axes of decoding strategy, data volume, and augmentation sophistication. The paper converts an abstract performance gap into a tractable research roadmap. The analysis establishes that the gap is not monolithic and not attributable to any single deficiency. The proposed model’s encoder quality approaches the SOTA level, a direct consequence of ImageNet pretraining. The deficit is concentrated in the decoder, which makes no explicit use of language structure, in the training corpus, which is three to four times smaller than SOTA benchmarks, and in the augmentation pipeline, which applies only linear affine transforms where elastic distortions are needed.

The generalization gap observable in the learning curves, training loss of 0.51 against a validation plateau of 1.61, a discrepancy of 1.10 nats is not noise. It is a measurement of the degree to which the model has memorized the handwriting styles present in the training partition rather than learning script-invariant visual primitives. That plateau arrived at epoch 127, well before the maximum training budget, confirms that the bottleneck is statistical, not computational. More data, or better synthetic diversity, would shift the plateau rightward and downward.

What this baseline affords the community is a reproducible, theoretically transparent reference point built from standard, publicly available components. Any subsequent system that adds elastic augmentation, beam search with an external language model, or an autoregressive decoder can now measure its incremental gain against a known CTC floor rather than against a heterogeneous collection of prior results trained under unstated conditions.

7. Future Works

The results reported in this chapter show that a modified ResNet34-BiLSTM-CTC architecture can learn unconstrained handwritten Urdu text lines, but they also show that this architecture is still far from the current best results in the field. The model reached a validation CER of 32.38%, which is an improvement over earlier shallow CTC-based approaches, but it remains significantly behind attention-based and Transformer-based systems. This gap should not be interpreted as a failure of the proposed method. Rather, it identifies the limits of a CTC-only recognition pipeline trained on a relatively small amount of handwritten Urdu data.

The experiments in this chapter show that the modified ResNet34-CRNN system can learn unconstrained Urdu text lines, but they also show where the method remains weak. The remaining errors are not caused by one factor only. Some arise from limited handwriting variation in the training data, some from visually similar characters that differ mainly by dot placement, and others from the limited linguistic context available to a CTC-only decoder.

Future research should therefore treat the current system as a baseline and examine which of these limitations has the largest practical effect on recognition accuracy.

7.1. Non-Linear Data Augmentation

Simard et al. [25] tested elastic distortion on the MNIST benchmark. They report that their results gave better generalization gains than any network architecture variation they tested, including two larger fully convolutional networks. They computed their distortion field as follows: The image deformations were created by first generating random displacement fields, that is $∆$ x (x, y) = rand(−1, +1) and $∆$ y (x, y) = rand(−1, +1), where rand(−1, +1) is a random number between -1 and +1, generated with a uniform distribution. The fields $∆$x and $∆$y are then convolved with a Gaussian of standard deviation σ (in pixels). If σ is large, the resulting values are very small because the random values average 0. If normalized, the displacement field (to a norm of 1), the field is then close to constant, with a random direction. If σ is small, the field looks like a completely random field after normalization for intermediate σ values, the displacement fields look like elastic deformation, where σ is the elasticity coefficient. The displacement fields are then multiplied by a scaling factorαthat controls the intensity of the deformation.

Elastic transformations can conduct a variety of non-linear deformations, such as stretching one ligature, squishing a neighbour, tilting the local writing line, and more, whereas a standard affine transform moves every point linearly. Wigington et al. [26] introduced two data augmentation and normalization techniques, which, used with a CNN-LSTM, significantly reduce Word Error Rate (WER) and Character Error Rate (CER) beyond best-reported results on handwriting recognition tasks. They applied a novel profile normalization technique to both word and line images, augmented existing text images using random perturbations on a regular grid and applied their normalization and augmentation to both training and test images. Their approach achieves low WER and CER over hundreds of authors, multiple languages and a variety of collections written centuries apart. Image augmentation in this manner achieves state- of-the-art recognition accuracy on several popular handwritten word benchmarks.

7.2. CTC with Shallow-Fused External Language Model

The Gulcehre et al. [27] explored two strategies on leveraging abundant amount of monolingual data for neural machine translation. They observe improvements by both combining scores from neural language model trained only on target monolingual data with neural machine translation model and fusing hidden states of these two models. They obtained up to 2 BLEU improvement over hierarchical and phrase-based baseline on low-resource language pair, Turkish→ English. Gulcehre et al. [27] into the following formula at decoding time.

$log p_{fused}(l | X)$ = $log p_{CTC}(l | X)$ + $\lambda ·\log p_{LM}\left(l\right)$This is also related to Graves and Jaitly [28] system which is based on a combination of the deep bidirectional LSTM recurrent neural network architecture and the CTC objective function. They modified objective function that trains the network to minimise the expectation of an arbitrary transcription loss function. This allowed a direct optimisation of the word error rate, even in the absence of a lexicon or language model. Their system achieved a word error rate of 27.3% on the Wall Street Journal corpus with no prior linguistic information, 21.9% with only a lexicon of allowed words, and 8.2% with a trigram language model. Combining the network with a baseline system further reduces the error rate to 6.7%. especially those involving characters whose strokes have close similarities that are hard to differentiate for a visual system such as “Pay” ( ث) is only visually different to “Tay” (ت) in having 1 additional stroke above.

7.3. Testing Attention Based Decoding for Urdu Script Sentences Decoder

The most substantial possible upgrade path is replacing the CTC-only recognition head with a CTC-attention hybrid architecture. Kim et al. [29] introduced method for end-to-end speech recognition to improve robustness and achieve fast convergence by using a joint CTC-attention model within the multi-task learning framework, thereby mitigating the alignment issue. An experiment on the WSJ and CHiME-4 tasks demonstrates its advantages over both the CTC and attention-based encoder-decoder baselines, showing 5.4-14.6% relative improvements in CER. It is derived from defining a combined loss as:

$L_{hybrid}$= α · $L_{CTC}+\left(1-\alpha \right)$ · $L_{Att}$ , $\alpha \in [0, 1]$CTC loss is $L_{CTC}$} and $L_{Att}$is the cross-entropy loss for the attention-based decoder-encoder system.

7.4. Using Nastaliq Font Representation Lines for Pretraining

Kang et al. In [30] proposed an unsupervised writer adaptation approach that can automatically adjust a generic handwritten word recognizer, fully trained with synthetic fonts, towards a new incoming writer. They experimentally validated using five different datasets, covering several challenges (i) the document source: modern and historic samples, which may involve paper degradation problems; (ii) different handwriting styles: single and multiple writer collections; and (iii) language, which involves different character combinations. Across these challenging collections, they showed the system can maintain its performance, thus, it provides a practical and generic approach to deal with new document collections without requiring any expensive and tedious manual annotation step.

7.5. Low-Rank Adaption for Writer Adaptive Fine Tuning

Bhunia et al [31] work on the assumption that there is always a new style that is drastically different, and that they will only have very limited data during testing to perform adaptation. This creates a commercially viable solution, being exposed to the new style, the model has the best shot at adaptation, and the few-sample nature makes it practical to implement. They achieve this via a novel meta-learning framework which exploits additional new-writer data via a support set, and outputs a writer-adapted model via single gradient step update, all during inference They leveraged on the important insight that there exist few key characters per writer that exhibit relatively larger style discrepancies and additionally propose to meta-learn instance specific weights for a character-wise cross-entropy loss, which is specifically designed to work with the sequential nature of text data. Their MetaHTR framework can be implemented on the top of most state-of-the-art HTR models. Their experiments show an average performance gain of 5-7% can be obtained by observing very few new style data. Hu et al [32] introduced Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. For GPT-3, LoRA can reduce the number of trainable parameters by 10,000 times and the computation hardware requirement by 3 times compared to full fine-tuning. LoRA performs on-par or better than fine-tuning in model quality on both GPT-3 and GPT-2, despite having fewer trainable parameters, a higher training throughput, and no additional inference latency 72 which means we can adapt the writing style to the low-resource Nastaliq and avoid catastrophic forgetting.