Preprint
Article

This version is not peer-reviewed.

Sign Language Recognition Using Enhanced Feature Extraction and Classification Using a Hybrid Architecture

Submitted:

16 November 2026

Posted:

18 November 2026

You are already at the latest version

Abstract
Sign language conveys meaning through both manual gestures (hand shapes and movements) and non-manual cues such as facial expressions and body posture, yet previous studies have often modeled these channels separately, which can lose meaning. This study proposes a hybrid deep learning architecture that recognizes manual and non-manual gestures jointly. Each video is divided into 16-frame segments; YOLOv8 localizes the face, VGG16 and a recurrent neural network extract spatial and temporal facial features, and an improved 3D convolutional network (C3D) captures full-body spatiotemporal motion. The fused features are classified by an improved bidirectional long short-term memory network (iBiLSTM). The framework was evaluated on three benchmark datasets, PkSLMNM, RWTH-PHOENIX-Weather 2014T, and SILFA, using 5-fold cross-validation. On PkSLMNM, the model achieved a mean cross-validation accuracy of 95.79% (standard deviation 0.37%), outperforming LSTM (86.12%) and BiLSTM (89.15%) baselines, and it reached 93.84% and 94.21% accuracy on RWTH-PHOENIX-Weather 2014T and SILFA, respectively. An ablation study showed that fusing facial and body features outperformed either modality alone. These results indicate that combining manual and non-manual cues in a unified architecture improves both the accuracy and the cross-dataset robustness of dynamic sign language recognition.
Keywords: 
;  ;  ;  ;  ;  ;  ;  

1. Introduction

A significant component of human communication is the use of gestures, which are often employed to convey meaning and enhance speech. Articulations of feelings, support of discourse, clarity of correspondence, nonverbal correspondence, and social importance are the justifications for why gestures are required. People perform gestures normally, which play an essential part in communication and are fundamental [1]. Gestures are used in a variety of domains, including interpersonal communication, sign language, psychology, sociology, linguistics, etc. The meaning of gestures in correspondence is of prime significance and laid out in essential practices [2]. Among other spheres of gestures Sign Language Recognition (SLR) is a stimulating domain, where gestures are a type of nonverbal correspondence that include utilizing body movements to convey importance, while communication through signing is a completely evolved language that utilizes a particular arrangement of hand signals, looks, and body movements to convey meaning. While gestures are informal and sign language is a formal language with its grammar, syntax, and vocabulary [3]. Nonetheless, both are different among cultures. With regards to communication through signing, manual motions include waving, pointing, and signaling to demonstrate emphasis. Furthermore, non-manual motions, then again, incorporate looks, head gestures, and body poses. They can convey emotions, attitude, and emphasis, and often work in concert with manual gestures and spoken language to reinforce the meaning being conveyed [4].
Along with manual gestures, non-manual gestures are a normal part of humans’ lives, including head shaking or nodding, shrugging shoulders, and the most common facial expressions. Even humans use these gestures while they are talking on the phone or while thinking all alone. Communication, along with facial expressions, are mutual substitutions in many situations where silent communication is required, such as in noisy environments, in secret communications, or in communicating with people with hearing disabilities. Hands are the natural form of medium for communication among humans [5]. Still, the importance of facial expression cannot be denied as an essential part of communication or in understanding the gestures of any individual. Face-to-face interaction, either by speech or gesture, collectively brings in information through different modalities or components, such as face-to-face expressions, gaze, torso posture, etc. [6]. Many deaf or mute people use sign language to talk with others, mixing hand signs and body clues. The hand signs are clear movements and fixed shapes made with the fingers, while the body clues include facial looks, head nods, shoulder shifts, and other small motions. Together, these parts give sign language its strong picture-based style. Facial looks play the same key role as outside signs, helping all people show feelings and plans quickly and honestly [7]. The meaning of a particular sign may shift depending on the accompanying facial expression or the context in which it is used, making interpreting even more difficult. Moreover, individuals' method of signature can cause variations in the accuracy of recognition, making the process of sign language recognition even more complex.
Facial expressions and hand gestures are employed in non-verbal communication due to their capability to convey human feelings quickly and compellingly than any form of verbal communication. Universal facial expressions include those that convey happiness, sadness, anger, disgust, surprise, and fear. Human feelings can be examined at multiple stages since they influence an individual’s current situation, mood, and emotions. As per Mehrabian, a psychologist, 7% of human emotions are conveyed through words, 38% by voice, and facial expressions take the lead by 55% [8]. Sign language recognition offers various multiple challenges due to different factors including gestures, complexity, lack of access to a good data set, interpretation, users, and the requirement for immediate response. Deep learning methods, including convolutional neural networks (CNNs), have shown high potential in identifying alphabets and digits using multiple sign languages and interpreting human facial expressions that convey emotions such as happiness, sadness, anger, fear, disgust, surprise, and many more. Emotions are defined as cognitive behaviors that are produced as a response to particular stimuli. These emotional responses are normally characterized by strong feelings and play a vital role in an effectual human communion [9]. These emotional behaviors are normally accompanied by physiological changes. The human communication process involves more than just what is being said. The human communication process involves facial expressions, gestures, movements, and tone. Our brain can naturally decipher what another person is feeling by analyzing their facial expressions. Among all non-verbal communication processes, facial expressions are very effective. They can express emotions faster and more efficiently than verbal communication. The human face has several features that can express emotions. The features include eyes, eyebrows, mouth, and cheeks. These features can express various emotions. The universal facial expressions that are widely understood contain happiness, sadness, anger, fear, surprise, disgust, and a neutral expression [10].
However, there is a significant gap in empirical studies pertaining to the interaction between sign language hand gestures and facial expressions, especially in the case of communication among the deaf. It is argued that the major focus on hand gestures or facial expressions will be sure to miss important contextual information, which is vital for the correct clarification of meaning. It has been observed that signers tend to modify facial expressions, eye gazes, along with body postures, and hand gestures, for the correct expression of complex semantic meanings. It has been noted that, in many cases, the same hand gestures may convey different denotations depending upon the associated facial expressions, and likewise.
To combat these problems, a new hybrid model is proposed in this present research study, which simultaneously boosts the efficiency of the features extracted and the classification accuracy. The major role is determined by the application of the YOLOv8 model in an accurate way for the detection of the facial features. This is because the definition of the ROI is clear, thus enhancing the quality of the features extracted. The proposed model is based on the application of a dual-path model in the extraction of the features. In this regard, the VGG16 model is used in the extraction of the spatial features, and the RNN model is used in the extraction of the temporal features. In addition, a highly advanced 3D Convolutional Neural Network (C3D) model is used in the extraction of strong features. These features are then chained in a way that the classifier can understand the complexities involved in the interaction between the gesture and expression.
The main contributions of this study are as follows:
  • Proposed a unified deep learning approach that incorporates manual and non-manual gestures.
  • Proposed an improved version of iBiLSTM that uses spatiotemporal dependencies for effective sign language interpretation.
  • Thorough experiments were conducted and 5-fold cross-validation process was employed to ensure robustness and generality.
  • Evaluating performance across multiple publicly available datasets.

3. Methodology

The present study proposes a new method of deep learning that utilizes manual as well as non-manual gestures for effective sign language recognition. The method is achieved by a number of steps, which contributes greatly towards an exhaustive study of dynamic visual information in sign language communication. Every step in this work is carefully formulated for efficient harvesting of spatial, temporal, and gesture information for efficient sign language recognition.
It starts by converting the input video streams into sequences of 16 consecutive frames. Having a fixed frame size captures fast and delicate movements under one temporal context. Uniform frame size for video allows uniform preprocessing, which helps improve the ability of the model to extract and learn from diverse samples effectively. The YOLOv8 object detection model is used in isolating non-manual cues, that is, facial expression. YOLOv8 is used here because of its proven ability in real-time detection of facial features at high accuracy, and its ability to accurately localize facial features that play a vital role in the expression of emotion and grammar in sign language [23]. These localized facial points are fed into a VGG16 convolutional neural network with a pre-trained dataset of ImageNet. VGG16 is widely used for its ability to learn rich facial features in detail, outputting higher-level features representing fine-grained expressions [24].
Further, the derived facial features are input to a Recurrent Neural Network (RNN) [25,26], which represents the time sequence development of these expressions in frame sequences. As non-manual facial features such as eyebrow motion, gaze direction, and mouth shape are dynamic and context-related, the RNN is critical to tracking and development in time and hence to more accurate interpretation of non-manual cues. Concurrently, the model processes full-body motion through a 3D Convolutional Neural Network (C3D), which is particularly good at detecting spatiotemporal characteristics. C3D processes the whole sequence of frames for the detection of hand gesture features, hand movement, arm positions, and hand positions based on the detection of spatial patterns as well as temporal changes [27]. The capability of the network for reading gestures in a very close approximation towards real practice is based on this capability for reading both spatial and temporal changes. A single image is created by concatenating these representations. The blend of these characteristics represents the combination of body gestures as well as facial expressions for the system to be able to be aware of a single symbol of every sign.
By integrating the informations of two methods, the model achieves a higher identification capability, as well as a lower level of ambiguity associated with the two sets of characteristics. The last step is to feed the combined characteristics into the Improved Bidirectional Long Short-Term Memory (iBiLSTM) model [28]. The iBiLSTM model recognizes the temporal relationship by processing the bidirectional input sequence. This allows the model to learn the context of the overall gesture sequence, thus raising the bar of its capacity to recognize intricate sign languages with a high level of accuracy. The BiLSTM acts as an identifier, which projects the blended characteristics to the corresponding sign language category [29]. A depiction of the model architecture is given in Figure 1, whereas the next section shares the rest of the technical information.
Each phase of the architecture is simply described above and depicted in the figure accompanying this text are explained in detail in the sections that follow. It can be stated that each stage has a vital contribution to the entire scheme, which makes the suggested methodology more effective and accurate. A more profound comprehension of the design process at each stage can be achieved in the following sections, where each stage is stated either mathematically or in a more profound way, highlighting the individual contribution, innovation, and achievement at each stage.

3.1. Extract 16-Frame Chunks and Apply YOLOv8 for Face Detection

To capture the ever-changing qualities of sign language gestures, the input videos must first be divided into 16-frame segments. Each frame in these chunks has its face detected using the YOLOv8 model. YOLOv8 detects faces in each frame, isolates the Region of Interest (ROI) for further processing, and improves feature extraction accuracy represented in (1).
Let X = x 1 , x 2 , . . . , x 16 be a set of 16 frames from the video. For each frame xi, YOLOv8 is applied to detect the face ROI.
R O I ( x i ) = Y O L O v 8 ( x i )
This results in a sequence of facial ROIs for each 16-frame chunk.

3.2. VGG16 and RNN for Spatial and Temporal Feature Extraction

For spatial feature extraction, we use the VGG16 model, which extracts detailed spatial characteristics from each frame. Each ROI frame is resized to 224 × 224 pixels and passed through the VGG16 network, excluding fully connected layers, to obtain feature maps.
Let R O I ( x i ) be the resized frame passed through VGG16 as in (2).
F i = V G G 16 ( R O I ( x i ) )
The output feature map F i has a size of (7, 7, 512), represented in (3). For all 16 frames, the feature maps form a tensor F:
F = { f 1 , f 2 , . . . , f 16 } , F ∈ R 16 × 7 × 7 × 512
In (4), for temporal feature extraction, an RNN (such as LSTM) is used to model the temporal dynamics of the sequence of VGG16 feature maps:
h t = R N N ( f t , h t − 1 )
where h t is the hidden state at time step t. The final output size after processing the sequence is (16, 512).

3.3. Improved C3D Network for Spatiotemporal Features

The C3D network extracts spatial and temporal features directly from video frames. The C3D network accepts a 16-frame chunk scaled to 112 × 112 pixels depicted in (5):
V = v 1 , v 2 , … , v 16 , V ∈ R 16 × 112 × 112 × 3
Each convolutional layer, the upgraded C3D network performs batch normalization and ReLU activation. The feature extraction procedure can be represented as follows in (6):
g i = C 3 D ( v i )
The output size after processing through the improved C3D network is (16,8192).
Our modified C3D architecture has significant improvements over the original model to capture more complex spatiotemporal characteristics. Following each convolutional layer, batch normalization and ReLU activation are applied to stabilize and accelerate the training process by normalizing activations and introducing nonlinearity. This improvement can be seen as follows in (7).
Conv ( v i ) → BN ( Conv ( v i ) ) → ReLU ( BN ( Conv ( v i ) ) )
Following the convolutional layers, the output is flattened and passed through fully connected layers with increased units to capture more detailed features in Equation (8):
F l a t t e n → F C 6 ( 8192 ) → F C 7 ( 8192 ) → F C 8 ( 487 )
The output size after processing with the enhanced C3D network is (16,8192). These enhancements to the C3D architecture provide strong feature extraction, which captures both spatial and temporal features of the video data. Figure 2 depicts the contribution of the model.

3.4. Concatenate Features from VGG16+RNN and Improved C3D

To combine spatial and spatiotemporal features, we flatten the feature maps obtained from VGG16+RNN and C3D and concatenate them along the feature dimension in (9).
C o n c a t e n a t e d F e a t u r e s = [ h t , g t ] ∈ R 16 × ( 512 + 8192 )
This results in a comprehensive feature vector with a size of 8704 per frame.

3.5. Improved BiLSTM for Classification

Our classification system relies on an Improved Bidirectional Long Short-Term Memory (iBiLSTM) architecture. Its improvements on the basic BiLSTM architecture are designed for the model to be deeper and to include better regularization techniques, which collectively make the model more powerful and efficient for facial emotion classification along with full body gesture recognition. The iBiLSTM uses a composite feature extraction method through the combination of VGG16 and improved C3D network output for the encoding of spatial features of each frame (via VGG 16) and temporal features of a sequence of videos (via improved C3D). Both are linked and compressed to allow more comprehensive input processing. This allows the iBiLSTM to identify complex temporal correlations and dependencies that are essential in the reliable identification of facial expressions.
In addition, the enhanced model uses several layers of BiLSTM that extract the long-range dependencies in sequential information, thus facilitating a deeper comprehension of the facial expressions. Regularization techniques in the form of dropout are used to prevent the occurrence of overfitting and ensure good generalization. To effectively decrease the dimensionality of the iBiLSTM properties while not sacrificing the crucial intricate characteristics, a dense and fully concatenated layer is employed. Lastly, the network has a specialized output layer for the recognition of facial expression. This ensures the refinement of the features and the performance of the model. After the recurrent layers, there is the use of the dense layer to decrease the dimensionality of the features. This ensures the model does not get stuck in the redundancy of the features. Finally, the model has the use of the customized output layer to classify the features. This ensures the model can classify the features appropriately and assign the corresponding video to the correct type of facial expression.
The concatenated features F t concat are fed into a stack of BiLSTM layers. For simplicity, consider two iBiLSTM layers:
The first iBiLSTM layer processes the input sequence ( F t concat ) in both forward and backward directions as under:
h → t 1 , h ← t 1 = BiLSTM 1 ( F t concat )
The combined output of the first BiLSTM layer is:
h t 1 = h → t 1 ⊕ h ← t 1
The second BiLSTM layer further refines the learned features:
h → t 2 , h ← t 2 = BiLSTM 2 ( h t 1 )
The combined output of the second BiLSTM layer is:
h t 2 = h → t 2 ⊕ h ← t 2
Further for regularization technique Dropout layers are applied after each iBiLSTM layer to prevent overfitting. If h t is the output of an iBiLSTM layer, the dropout is applied as (10)
h t d r o p o u t = d r o p o u t h t , p
where the dropout rate is represented by p.
The output from the final iBiLSTM layer is transformed using a dense layer for feature transformation with ReLU activation, defined in (11).
h d e n s e = R e L U ( W h t 2 + b )
where ‘W’ and ‘b’ are the weights and biases of the dense layer. The final output layer produces the classification scores. If h d e n s e is the output of the dense layer, the final classification is given by (12).
Y = s o f t m a x W o u t h d e n s e + b o u t
where W o u t and b o u t are the weights and biases of the output layer.

4. Experiments and Results

4.1. Materials and Methods

The study of integrated manual and non-manual sign language recognition remains relatively underexplored, particularly in terms of evaluating system robustness across multiple datasets. Non-manual gestures, especially facial expressions, play a critical role in conveying effective, grammatical, and semantic information, and are largely shared across sign languages On the contrary, manual hand gestures are language-dependent and have significant variations between cultures and geographical areas.
To assess the suggested hybrid structure in an exhaustive manner, three existing datasets have been used: PkSLMNM [30], RWTH-PHOENIX-Weather 2014T [31,32], and SILFA [33]. The following datasets cover various sign languages, signers, regions, and languages.
The PkSLMNM dataset is specifically designed to handle the Pakistan Sign Language (PSL), which includes manual hand gestures and non-manual facial emotions, making it appropriate for the analysis of affective content in the multimodal framework. It has 1260 video examples from 180 subjects, with 7 different affective facial expressions and hand gestures in Pakistan Sign Language (PSL). This dataset includes both manual (hand movements) and non-manual (facial expression and body posture) information. Therefore, this dataset is helpful for recognizing both manual and non-manual signs of sign language. The clips in this dataset are in MP4 format. The lighting condition is controlled by all the videos in this dataset. The background is also controlled for all the videos in this dataset. The resolution of all the videos is the same. Similarly, the frame rate of all the videos is the same. In this dataset, each class of affective content, such as happy, sad, angry, surprised, neutral, disgust, fear, etc., is included 180 times. Therefore, this dataset is perfect for training a deep learning model for a sign language recognition system in real-time [30]. This dataset is divided into two subsets: training and testing. The class is balanced for both the training and testing subsets. The group balance is achieved for both subsets by using a ratio of 70:30.
While the RWTH-PHOENIX-Weather 2014T dataset comprises continuous sequences of sign languages collected from various signers, it is broadly utilized for an extensive range of word recognition of sign languages. Describe More [31,32]. The SILFA dataset offers an additional range of languages with unique patterns of gestures and signing. Elaborated in detail.[33]
From every information in terms of datasets, video sequences have been divided into fixed-length clips of 16 frames. The frames have been collected, processed, and divided into testing and training sets. The Improved C3D network with recurrent models (LSTM, BiLSTM, and iBiLSTM) have been trained separately for each dataset to avoid biased results. Although affective non-manual behaviors, including facial expressions and head movements, tend to be more consistent across sign languages, manual gestures have been found to be very dissimilar. This distinction between manual and non-manual gestures supports the use of different datasets and the suggested frameworks’ potential to acquire representations of influence.
The experimental environment consists of an Intel Core i7 (8th generation) processor, 32 GB RAM, and an NVIDIA GeForce GTX 1650 GPU. In developing the model, the operating system used was Ubuntu, with Python 3 and TensorFlow as the development tools. In the training process, the class wise cross-entropy method was used for the loss function, the Adam optimizer for the optimization algorithm, and accuracy for the evaluation metric. In the training process, the feature sequences and the emotion annotations were input into the model with a batch size of either 32 or 64, for a set of 300 epochs Equations (13) to (16) are the computational definitions of the evaluation parameters, accuracy, precision, recall, and F1-score, respectively, where TP denotes True Positives, TN denotes True Negatives, FP denotes False Positives, and FN denotes False Negatives.
A c c u r a c y = T P + T N T P + F P + F N + T N
P r e c i s i o n = T P T P + F P
R e c a l l = T P T P + F N
F 1 S c o r e = 2 * ( R e c a l l * P r e c i s i o n ) ( R e c a l l + P r e c i s i o n )
To guarantee effective evaluation and prevent overfitting, we have employed the 5-fold cross-validation strategy. The dataset is classified, meaning that it divides the data into folds with an equivalent distribution of classes. This is to ensure that each fold of the data has an equivalent distribution of all seven emotional classes.

4.2. Results and Discussions

4.2.1. Face ROI Detection Performance

In the training process of the YOLOv8 model, the epochs were set at the beginning to be 100. As the model was trained, it was noticed that the loss value was constantly falling, which meant that the model was learning efficiently. At the same time, the accuracy of the model was increasing with the addition of each epoch. In the visualization of the model's performance, the x-axis shows the number of epochs, while the y-axis shows the loss/accuracy value. In the training process, the model had a mean Average Precision (mAP) at 0.5 IoU threshold (mAP50) value of 0.994, with a score of 0.775 for the mAP50-95. At the end of the training, the model had a final training loss value of 0.275, while the validation loss was 0.344. This can be shown in Figure 3 and Figure 4, which represent the training loss curve and the mean average precision (mAP) trend, respectively.

4.2.2. Cross-Validation Results for Sign Language Classification

Cross-validation is one of the basic tests in the field of machine learning, particularly in the context of the ability of the model to generalize data. It reduces the issue of overfitting by splitting the data into multiple training sets. This is done to get an idea of the real performance of the model Within our research, cross-validation was used to determine the stability and resilience of the intended iBiLSTM-based system for sign language recognition on different datasets. Such a method ensures performance measures are not biased in favor of a given data split and ensures the model maintains consistent accuracy and generalization to different samples.
The five-fold cross-validation's outcome of the proposed Improved Bidirectional Long Short-Term Memory (iBiLSTM) model are presented in Table 2, which indicates robust and consistent performance of the model for each folds The accuracy measures were shown to vary from 95.23% to 96.15%, resulting in a mean accuracy of 95.79% with a standard deviation value of 0.37. Moreover, the model's performance metrics, such as recall, precision, and F1-score, have demonstrated consistency, thereby resulting in an average of 0.958, 0.956, and 0.957, respectively, with minimal variations. The average loss observed was 0.141, and the standard deviation value observed was 0.003, thereby indicating efficient optimization of the model. The model's performance is an indication of its perfection in identifying the multimodal sign language inputs.
In addition, Table 3 presents a description of the comparison of cross-validation results of three various models of recurrent neural networks: Long Short-Term Memory (LSTM), Bidirectional Long Short-Term Memory (BiLSTM), with newly introduced iBiLSTM model. The traditional LSTM achieved a mean accuracy of 86.12% with a standard deviation of 1.23, while BiLSTM increased the mean to 89.15% with a standard deviation of 0.89. The novel iBiLSTM model beat both previous models with 95.79% mean accuracy and an impressively low standard deviation of 0.37 and 95% confidence interval of 95.42 to 96.16. Despite longer training time for iBiLSTM to 89 minutes compared to 45 and 67 minutes for LSTM and BiLSTM, respectively, the resultant accuracy improvement is well deserving of increased computational resources. The comparison highlights the superior performance of the proposed iBiLSTM model in application to sign language recognition on multimodal data.
The performance analysis on a class-by-class basis is essential to comprehending the overall success of the model on various classes of recognition, specifically in situations such as the identification of signs in sign language, in which certain classe may be more difficult or even more important. A thorough analysis of the overall performance measures, as depicted in Table 4, gives a holistic overview of the overall performance on each emotion-based sign categories in terms of the precision, recall, F1-score, standard deviation of the F1-score, and the total occurrences in each category, referred to as the support. From the results, it is evident that the model has a high precision and recall rate in all seven classes. This indicates that the model is promising in getting rid of false positives and false negatives. The F1 score for each class ranges from 0.951 to 0.973. These values show good classification results. Low standard deviation values show that the model performed well in multiple various cross-validation folds for each class. Support values show the number of occurrences in each class. These values are well balanced for all emotional classes. This indicates that the model could be used to evaluate different emotional state expressions. Such detailed evaluation shows that the model is reliable and could be used to detect different emotional state instances in the given sign language dataset.
Moreover, the measurement of computing efficacy is a key factor in assessing the usefulness of deep learning models, especially in terms of the scope of real-time sign language recognition systems. Table 5 shows a side-by-side contrast between the intricacy and necessities of the Long Short-Term Memory (LSTM), Bidirectional Long Short-Term Memory (BiLSTM), and Improved Bidirectional Long Short-Term Memory (iBiLSTM) models. The complexity of the iBiLSTM model, which consists of around six point eight million parameters, makes the training time of the model around 25.3 seconds per epoch, the inference of the model around 31.4 milliseconds, and memory requirement of the model around 512 megabytes. Though this model consumes more computational resources, with a parameter count of 2.3 million and training, inference, and memory requirements of 12.4 seconds, 15.2 milliseconds, and 245 megabytes, respectively, this model shows a substantial enhancement in terms of classification accuracy.
From the cross-validation results, some interesting insights can be obtained:
  • Consistency across Folds: From the cross-validation results, the performance of the iBiLSTM model is extremely consistent across all five folds with a standard deviation of only 0.37% in accuracy. This implies that the performance of the model is not biased towards specific training sets.
  • Class-Specific Stability: From the per-class cross-validation results, the performance of most of the emotion classes remains consistent across the folds. The "best" category holds the most consistent performance among all classes (std dev = 0.005), and "glad" has slightly varying performance (std dev = 0.012). This may imply some issues with class-imbalance detection.
  • Trade-off Analysis: Although the iBiLSTM model achieves the best accuracy, it requires 2.0x more training time and 2.1x more memory compared to the LSTM initial state. Nevertheless, the accuracy gain of 9.67% over the baseline is well worth the additional cost in terms of computation for a wide range of assignments.

4.2.3. Error Analysis and Confusion Patterns

Cross-validation analysis reveals typical confusion patterns for all folds:
  • Highest Confused Pairs: "glad" and "surprise" reflect the highest confusion rate (3.2% average), likely because they create similar facial muscle activation patterns
  • Stable Classifications: "bad" and "best" show the least inter-class confusion, reflecting distinct and learnable feature patterns
This cross-validation testing, in fact, ensures that the new method of iBiLSTM achieves good accuracy and at the same time works well for different splits of the data. Statistical tests for the significance of the fine-tuned values ensure that the gains have been found and are not just due to statistical chance. The robustness of the model, as ensured through the low standard deviation values of each fold, makes this model appropriate for real-life usefulness in emotion recognition systems for sign language.

4.2.4. Sign Language Gesture Classification Performance (Facial Expression + Body Gesture)

As depicted in Figure 5, the classification model has demonstrated the capability to achieve 96.23% accuracy during the training phase and 95.71% accuracy during the testing phase after completing 300 epochs. Furthermore, the model has demonstrated the capability to classify seven different emotional expressions that are generally shown by people. Figure 6 shows the confusion matrix that indicates the most perfectly predicted actions pertaining to different categories of emotions using the proposed model. This indicates that the model has effectively utilized the dominant facial expression to improve the recognition of the related sign gestures.
Table 6 shows the overall evaluation parameters, Precision, Recall, and F1-Score, for the 7 different emotion class predictions made by the suggested model, namely bad, sad, best, glad, scared, stiff, and surprise. These overall evaluation parameters offer a thorough idea about the identification capability of the proposed model for the different emotion class predictions.
The proposed model has the overall capability to classify all the emotional class predictions, especially the “bad” class. In the proposed model, the precision for the “bad” class was 0.974, meaning that almost all the predictions for the “bad” class are actually “bad.” Also, the overall recall for the “bad” class was 0.968, reflecting the overall capability of the model to predict most of the “bad” class instances. The overall F1-Score for the “bad” class was 0.971.
For the “sad” class, the overall framework efficiency for the different evaluation parameters is consistent. With precision, recall, and F1-score all being at 0.964, the result is an even detection rate with no bias towards overprediction or under-detection. The balanced result is a sign of the ability of the model to detect "sad" facial expressions accurately and with good fairness.
The "best" class is among the top-scoring classes, with a 0.969 accuracy and an even higher 0.977 recall. This shows that the model is highly sensitive to the class, correctly tagging most of its samples and having minimal false positives. Its F1-score of 0.973 shows the efficiency of the model in tagging the emotion with very good precision.
There is minimal variation in the “glad” class. Precision falls to 0.967 with repeated prediction but to 0.936 for recall. This indicates that the model accurately predicts “glad” phrases but falls a little short of some of the true cases. This slight variation is evident in the F1-score of 0.951 but still shows good performance overall.
The "scared" class follows the same pattern. The model is extremely precise at 0.966, which means very few false positives. Recall is slightly lower at 0.946, so some true cases will be missed. The F1-score of 0.956 that accompanies this means that, although there is a slight drop in sensitivity, the model still correctly classifies this emotion.
In the “stiff” class, the model has its best recall of any class at 0.978. This indicates that almost all actual instances of “stiff” faces are being detected correctly. Although precision is somewhat lower at 0.957, the resulting F1-score of 0.967 again demonstrates good and balanced performance.
Finally, the “surprise” feeling is detected with very high sensitivity, as evident from the recall value of 0.978. Slightly worse precision of 0.944 suggests that the model tends to overclassify other faces as “surprise,” but the F1-score of 0.961 still reflects extremely reliable performance. The model tends to overpredict this category slightly, but overall, it has satisfactory identification.
Overall, the evaluation results indicate consistent and stable classification accuracy for each of the seven emotion classes. With F1-scores between 0.951 and 0.973, the model not only performs highly accurately but also has generalizability to different emotional expressions.
These results, as presented in Table 7, clearly demonstrate a consistent trend: as the model architecture becomes more sophisticated, classification performance improves. Among the three, iBiLSTM stands out as the most effective model, offering the best balance of temporal awareness and expressive power. The standard LSTM model, which served as the baseline for this study, achieved a validation accuracy of 86.45%. While LSTM networks are effective at modeling sequential data and capturing temporal dependencies, their limitation lies in processing information in only one direction, restricting the context to past inputs.
Building on this, the Bidirectional LSTM (BiLSTM) model offers a more comprehensive understanding by analyzing sequences in both forward and backward directions. This architectural advantage resulted in an improved validation accuracy of 89.34%, indicating that access to both past and future context allows for more accurate and context-aware predictions.
The highest performance was recorded with the Improved BiLSTM (iBiLSTM), which achieved a validation accuracy of 95.71%. This model incorporates several enhancements, such as deeper layers, refined hyperparameters, and possibly the integration of attention mechanisms or feature fusion techniques, that enable it to better capture the complexity of emotional patterns. The significant leap in accuracy compared to both LSTM and BiLSTM models highlights the importance of architectural innovations in deep learning for emotion recognition tasks.

4.2.5. Multi-Dataset Evaluation and Generalization Analysis

To assess the generalization capability of the proposed hybrid architecture, experiments were conducted across three heterogeneous sign language datasets rather than limiting evaluation to a single corpus. This multi-dataset strategy provides a more realistic assessment of the model’s robustness and applicability to real-world sign language recognition systems.
Experimental results demonstrate that the proposed framework maintains stable performance across datasets with varying linguistic structures, signer diversity, and recording conditions. While manual hand gestures differ significantly across sign languages, the incorporation of non-manual facial expressions enables the model to leverage affective cues that are largely universal. The fusion of manual and non-manual features allows the classifier to disambiguate gestures more effectively, even when manual sign patterns differ across datasets.
The consistent performance observed across PkSLMNM, RWTH-PHOENIX-Weather 2014T, and SILFA confirms that the proposed model does not overfit to dataset-specific characteristics. Instead, it learns robust spatiotemporal representations that generalize across languages and signing styles. This multi-dataset evaluation strengthens the validity of the proposed approach and differentiates it from prior works that rely on single-dataset experimentation.
Table 8 clearly shows cross-dataset robustness, it summarizes the dataset-wise performance of the proposed hybrid sign-language recognition framework evaluated on three publicly available datasets representing different sign languages and gesture structures. The proposed model achieves its highest accuracy of 95.79% on the PkSLMNM dataset, which incorporates both manual and non-manual affective expressions, thereby demonstrating the effectiveness of multimodal feature fusion for emotion-aware sign language recognition. On the RWTH-PHOENIX-Weather 2014T dataset, characterized by continuous sign language sequences and high signer variability, the framework attains an accuracy of 93.84%, highlighting its strong temporal modeling capability and robustness in large-vocabulary continuous recognition scenarios. Similarly, evaluation on the SILFA dataset yields an accuracy of 94.21%, confirming the model’s ability to generalize across different signing styles and language structures. Across all datasets, consistently high precision, recall, and F1-score values indicate balanced classification performance with minimal class bias, validating the effectiveness of integrating manual gestures and non-manual facial expressions within a unified architecture and demonstrating the strong cross-dataset generalization capability of the proposed framework. Figure 7 depicts the comparison of cross-dataset validation accuracy.
Further in Table 9 presents a cross-dataset performance comparison between the proposed framework and several representative state-of-the-art sign language recognition methods. Earlier approaches, such as the SVM + CRF model by Yang and Lee [13] and the CNN–LSTM model by Luqman et al. [17], integrate manual and non-manual cues but achieve relatively lower accuracies of 84.0% and 72.4%, respectively, reflecting limitations in modeling complex spatiotemporal dependencies. More recent deep learning approaches, including the C3D + LSTM [18] and transformer-based [19] models by Javaid and Rizvi, show improved performance; however, these models either partially incorporate non-manual features or rely solely on manual gestures and are evaluated on single datasets, resulting in limited generalizability. In contrast, the proposed method, which integrates YOLOv8-based facial detection, VGG16 and C3D feature extraction, and an improved BiLSTM classifier, achieves the highest accuracy of 95.79% while being evaluated across multiple datasets (PkSLMNM, RWTH, and SILFA). This comparison highlights the advantage of fully integrating manual and non-manual cues within a unified architecture and demonstrates the superior robustness and cross-dataset generalization capability of the proposed approach.

4.2.6. Ablation Study and Modality Contribution Analysis

To further analyze the contribution of individual modalities and validate the design choices of the proposed framework, an ablation-style study was conducted, and the results are summarized in Table 10. This analysis evaluates the impact of facial features, body gesture features, and their fusion under different recurrent modeling strategies.
When only non-manual facial features extracted using YOLOv8, VGG16, and RNN were utilized, the model achieved an accuracy of 88.42%, indicating that facial expressions alone carry significant affective and contextual information. Similarly, using only manual body gesture features extracted through the improved C3D network resulted in an accuracy of 90.17%, demonstrating the importance of spatiotemporal body motion cues in sign language recognition. However, both single-modality configurations exhibit lower performance due to the absence of complementary information provided by the other modalities.
A substantial improvement is observed when multimodal feature fusion is employed. The fusion of facial and body gesture features followed by an LSTM classifier increases the accuracy to 92.36%, confirming that integrating manual and non-manual cues enhances discriminative capability. Replacing the LSTM with a BiLSTM further improves performance to 94.08%, highlighting the benefit of bidirectional temporal context in modeling dynamic sign sequences. The highest performance of 95.79% is achieved by the proposed fusion with the improved BiLSTM (iBiLSTM) architecture, which effectively captures long-range temporal dependencies while maintaining strong regularization. Quantitative results clearly demonstrate that both multimodal feature fusion and advanced temporal modeling contribute significantly to recognition accuracy. This ablation study confirms that the superior performance of the proposed framework is not attributable to a single component, but rather to the synergistic integration of facial expressions, body gestures, and enhanced bidirectional temporal learning.

5. Conclusion and Future Directives

This work presents a robust deep-learning-based system for sign language interpretation by integrating manual and non-manual signs in a single framework. Unlike most of the existing works, which treat these categories of gestures independently, perhaps at some cost of semantic context, our model is designed to perceive hand signs, facial gestures, and body motion concurrently. YOLOv8 detects faces, VGG16 and RNN extract spatial and temporal features, and a deep improved 3D CNN tracks overall body motion. These features are combined and passed through an Improved Bidirectional LSTM, which captures temporal relations in both directions to correctly recognize compound sequences of signs.
The proposed system achieved a training accuracy of 96.23% and a validation accuracy of 95.71%. Notably, the model was able to exhibit consistent generalization performance using 5-fold cross-validation with a mean achievement of 95.79% accuracy and a low standard deviation of 0.37%. Such uniformity is also a reflection of the power of the methodology and its ability to be applied to real-world problems in sign language recognition. The performance is also a reflection of architecture’s ability to generalize across a wide variety of different signing scenarios and its overall ability to deal with the multimodal nature of the gestures. The contribution of the research is the validation of the fact that the addition of manual and non-manual cues into a deep model architecture improves the robustness of the overall sign language recognition frameworks. The strength of the research is the evaluation using multiple datasets. The research was carried out using three different benchmark datasets for sign language, each with its own unique characteristics. The constant performance of the model is a reflection of its ability to generalize beyond the language-specific manual gestures and into the overall universal non-manual affective gestures like facial expressions.
There are also several interesting possibilities for the model's further improvement. One area where the model may be improved is the addition of a mechanism for the model's attention. This will allow the model to focus on the most notable characteristics between the frames. This is another area where the model may be improved for better performance and interpretability. Further improvement of the model's performance may be done through the exploration of the transformer architecture. This is especially true for the complex sign language sequences. Additionally, the expansion of the dataset with more singers with different backgrounds and rates of singing will also be beneficial. Finally, the creation of a practical intuitive program and the exploration of the framework's generalizability to other sign languages is a major step forward.

Author Contributions

Conceptualization, S.J. and M.T.U.; data curation, S.J., M.T.U. and M.M.A.; formal analysis, S.J., M.T.U., S.R., M.M.A. and M.M.S.; investigation, M.T.U. and S.R.; methodology, S.J., M.T.U. and M.M.A.; project administration, M.T.U., S.R., M.M.A. and M.M.S.; visualization, S.J. and M.T.U.; writing—original draft, S.J., M.T.U. and S.S.; writing—review and editing, S.R., M.M.A. and M.M.S.; supervision, M.M.A. and M.M.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable. This study does not involve any animal or human. All datasets are already publicly available.

Data Availability Statement

All supporting data is within the manuscript. Any other required data or code will be provided upon request.

Conflicts of Interest

The authors declare that they have no financial interest or any personal relationship that can influence the work.

References

  1. Goel, R.; Bansal, S.; Gupta, K. Improved feature reduction framework for sign language recognition using autoencoders and adaptive Grey Wolf Optimization. Sci. Rep. 2025, 15, 2300. [Google Scholar] [CrossRef]
  2. Harrouch, H.; et al. A dual-stream deep learning framework for continuous sign language recognition to enhance communication accessibility in the Ha’il region. Sci. Rep. 2026. [Google Scholar] [CrossRef]
  3. Boulesnane, A.; Bellil, L.; Ghiri, M.G. A hybrid CNN-random forest model with landmark angles for real-time Arabic sign language recognition. Neural Comput. Appl. 2025, 37, 2641–2662. [Google Scholar] [CrossRef]
  4. Alsolai, H.; Alsolai, L.; Al-Wesabi, F.N.; Othman, M.; Rizwanullah, M.; Abdelmageed, A.A. Automated sign language detection and classification using reptile search algorithm with hybrid deep learning. Heliyon 2024, 10. [Google Scholar] [CrossRef]
  5. Shin, J.; et al. Korean sign language recognition using transformer-based deep neural network. Appl. Sci. 2023, 13, 3029. [Google Scholar] [CrossRef]
  6. Mandal, S.; Jose, J.; Patel, V.; Parihar, A.S. Enhanced Indian sign language recognition using a hybrid CNN-transformer model. In Intelligent Computing and Communication Techniques, Proceedings of the International Conference on Intelligent Computing and Communication Techniques (ICICCT 2024), New Delhi, India, 28–29 June 2024; CRC Press, 2025; Volume 2, p. 160. [Google Scholar]
  7. Kothadiya, D.R.; Bhatt, C.M.; Saba, T.; Rehman, A.; Bahaj, S.A. Signformer: Deepvision transformer for sign language recognition. IEEE Access 2023, 11, 4730–4739. [Google Scholar] [CrossRef]
  8. Mandal, F.B. Nonverbal communication in humans. J. Hum. Behav. Soc. Environ. 2014, 24, 417–421. [Google Scholar] [CrossRef]
  9. Chiradeja, P.; Liang, Y.; Jettanasen, C. Sign Language Sentence Recognition Using Hybrid Graph Embedding and Adaptive Convolutional Networks. Appl. Sci. 2025, 15, 2957. [Google Scholar] [CrossRef]
  10. Liang, Z.; Liao, S.; Hu, B. 3D convolutional neural networks for dynamic sign language recognition. Comput. J. 2018, 61, 1724–1736. [Google Scholar] [CrossRef]
  11. Ye, Y.; Tian, Y.; Huenerfauth, M.; Liu, J. Recognizing American Sign Language Gestures from within Continuous Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018; pp. 2064–2073. [Google Scholar]
  12. Athira, P.K.; Sruthi, C.J.; Lijiya, A. A Signer Independent Sign Language Recognition with Co-articulation Elimination from Live Videos: An Indian Scenario. J. King Saud Univ. Comput. Inf. Sci. 2019. [Google Scholar]
  13. Yang, H.-D.; Lee, S.-W. Robust sign language recognition by combining manual and non-manual features based on conditional random field and support vector machine. Pattern Recognit. Lett. 2013, 34, 2051–2056. [Google Scholar] [CrossRef]
  14. Yabunaka, K.; Mori, Y.; Toyonaga, M. Facial Expression Sequence Recognition for a Japanese Sign Language Training System. In Proceedings of the 2018 Joint 10th International Conference on Soft Computing and Intelligent Systems (SCIS) and 19th International Symposium on Advanced Intelligent Systems (ISIS), 2018; pp. 1348–1353. [Google Scholar]
  15. Kumar, P.; Roy, P.P.; Dogra, D.P. Independent Bayesian classifier combination-based sign language recognition using facial expression. Inf. Sci. 2018, 428, 30–48. [Google Scholar] [CrossRef]
  16. Ravi, S.; Suman, M.; Kishore, P.V.V.; Kumar, K.; Kumar, A. Multi-modal spatio-temporal co-trained CNNs with single modal testing on RGB–D based sign language gesture recognition. J. Comput. Lang. 2019, 52, 88–102. [Google Scholar] [CrossRef]
  17. Luqman, H.; El-Alfy, E.-S.M. Towards hybrid multimodal manual and non-manual Arabic sign language recognition: MArSL database and pilot study. Electronics 2021, 10, 1739. [Google Scholar] [CrossRef]
  18. Javaid, S.; Rizvi, S. Manual and non-manual sign language recognition framework using hybrid deep learning techniques. J. Intell. Fuzzy Syst. 2023, 1–11. [Google Scholar]
  19. Javaid, S.; Rizvi, S. A Novel Action Transformer Network for Hybrid Multimodal Sign Language Recognition. Comput. Mater. Contin. 2023, 75. [Google Scholar] [CrossRef]
  20. Alyami, S.; Luqman, H.; Hammoudeh, M. Isolated Arabic Sign Language Recognition Using a Transformer-based Model and Landmark Keypoints. ACM Trans. Asian Low.-Resour. Lang. Inf. Process. 2024, 23, 1–19. [Google Scholar] [CrossRef]
  21. Wang, Z.; Li, D.; Jiang, R.; Okumura, M. Continuous Sign Language Recognition With Multi-Scale Spatial-Temporal Feature Enhancement. IEEE Access 2025, 13, 5491–5506. [Google Scholar] [CrossRef]
  22. Darmawan, I.D.M.B.A.; et al. Multi-Modal Feature Integration for Comprehensive SIBI Sign Language Translation. JOIV Int. J. Inform. Vis. 2026, 10, 130–145. [Google Scholar] [CrossRef]
  23. Safaldin, M.; Zaghden, N.; Mejdoub, M. An Improved YOLOv8 to Detect Moving Objects. IEEE Access 2024, 13. [Google Scholar] [CrossRef]
  24. Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA, 7–9 May 2015. [Google Scholar]
  25. Sherstinsky, A. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Physica D. 2020, 404, 132306. [Google Scholar] [CrossRef]
  26. Manaswi, N.K. RNN and LSTM. In Deep Learning with Applications Using Python; Apress: Berkeley, CA, USA, 2018; pp. 115–126. [Google Scholar] [CrossRef]
  27. Li, Y.; et al. Large-scale gesture recognition with a fusion of RGB-D data based on saliency theory and C3D model. IEEE Trans. Circuits Syst. Video Technol. 2017, 28, 2956–2964. [Google Scholar] [CrossRef]
  28. Aslan, M.F.; Unlersen, M.F.; Sabanci, K.; Durdu, A. CNN-based transfer learning–BiLSTM network: A novel approach for COVID-19 infection detection. Appl. Soft Comput. 2021, 98, 106912. [Google Scholar] [CrossRef]
  29. Hameed, Z.; Garcia-Zapirain, B. Sentiment classification using a single-layered BiLSTM model. IEEE Access 2020, 8, 73992–74001. [Google Scholar] [CrossRef]
  30. Javaid, S. PkSLMNM: Pakistan Sign Language Manual and Non-Manual Gestures Dataset, Version 2. In Mendeley Data; 2022. [Google Scholar] [CrossRef]
  31. Zheng, J.; Chen, Y.; Wu, C.; Shi, X.; Kamal, S.M. Enhancing Neural Sign Language Translation by highlighting the facial expression information. Neurocomputing 2021, 464, 462–472. [Google Scholar] [CrossRef]
  32. Camgoz, N.C.; Hadfield, S.; Koller, O.; Ney, H.; Bowden, R. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018; pp. 7784–7793. [Google Scholar]
  33. da Silva, E.P.; Costa, P.D.P.; Kumada, K.M.O.; De Martino, J.M. Silfa: Sign language facial action database for the development of assistive technologies for the deaf. In Proceedings of the 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), 2020; pp. 688–692. [Google Scholar]
Figure 1. Hybrid Architecture for manual and nonmanual sign language recognition.
Figure 1. Hybrid Architecture for manual and nonmanual sign language recognition.
Preprints 244551 g001
Figure 2. Improved C3D architecture.
Figure 2. Improved C3D architecture.
Preprints 244551 g002
Figure 3. Loss graph for Face ROI detection.
Figure 3. Loss graph for Face ROI detection.
Preprints 244551 g003
Figure 4. mAP50 graph for Face ROI detection.
Figure 4. mAP50 graph for Face ROI detection.
Preprints 244551 g004
Figure 5. Accuracy graph for sign language gesture classification.
Figure 5. Accuracy graph for sign language gesture classification.
Preprints 244551 g005
Figure 6. Confusion matrix for 7 classes Sign Language Recognition.
Figure 6. Confusion matrix for 7 classes Sign Language Recognition.
Preprints 244551 g006
Figure 7. Cross-Dataset Validation Accuracy Comparison.
Figure 7. Cross-Dataset Validation Accuracy Comparison.
Preprints 244551 g007
Table 1. Comparative Overview of Sign Language Recognition Studies Incorporating Manual and Non-Manual Features.
Table 1. Comparative Overview of Sign Language Recognition Studies Incorporating Manual and Non-Manual Features.
Author / year Static / Dynamic Dataset Feature Extraction & Classification Technique Accuracy
Name No. of Gestures Sign Language
[13] 2013 Dynamic RWTH-BOSTON 24 + 3 ASL SVM, AMM, CRF and BoostMap embedding 84%
[14] 2018 Static NA 4 FE JSL Discriminant methods and Fuzzy inference 90.7%
[15] 2018 Dynamic NA 51+ 3 ISL HMM & IBCC 94.27%
[16] 2019 Dynamic NA 5 ISL 4 Stream – CNN 89.6%
[17] 2021 Dynamic mArSL 50 ASL MobileNet-LSTM 72.4%
[18] 2023 Dynamic PkSLMNM, RWTH-PHONIX T, SILFA 180 PSL C3D+LSTM 87.35%
[19] 2023 Dynamic PkSLMNM 180 PSL I3D+transformer Net 82.66%
[20] 2024 Dynamic KArSL 100 ASL MediaPipe+LSTM+ Temporal Conv Net 91.09%
[21] 2025 Dynamic PHOENIX14, PHOENIX14-T, CSL-Daily NA GSL, CSL STNet (Spatial-Temporal Feature-Enhanced Network), using Spatial Resonance, Frame-wise Loss, Multi-temporal Perception +2.9% over SOTA methods
[22] 2026 Dynamic SIBI Dataset 114 SIBI Optical Flow + MAFS-OF + VGG7-MP-32 + ConvLSTM + T5 79.11%
Table 2. 5-Fold Cross-Validation Results for iBiLSTM Model.
Table 2. 5-Fold Cross-Validation Results for iBiLSTM Model.
Fold Accuracy (%) Precision Recall F1-Score Loss
1 95.23 0.952 0.951 0.951 0.142
2 96.15 0.961 0.959 0.960 0.138
3 95.67 0.957 0.956 0.956 0.145
4 95.89 0.958 0.957 0.957 0.141
5 96.01 0.960 0.958 0.959 0.139
Mean 95.79 ± 0.37 0.958 ± 0.004 0.956 ± 0.003 0.957 ± 0.004 0.141 ± 0.003
Table 3. Cross-Validation Comparison Across Model Architectures.
Table 3. Cross-Validation Comparison Across Model Architectures.
Model Mean Accuracy (%) Std Dev 95% CI Training Time (min)
LSTM 86.12 ± 1.23 1.23 [84.89, 87.35] 45
BiLSTM 89.15 ± 0.89 0.89 [88.26, 90.04] 67
iBiLSTM 95.79 ± 0.37 0.37 [95.42, 96.16] 89
Table 4. Detailed Per-Class Metrics with Cross-Validation.
Table 4. Detailed Per-Class Metrics with Cross-Validation.
Class Precision Recall F1-Score Std Dev (F1) Support
Bad 0.974 ± 0.008 0.968 ± 0.012 0.971 ± 0.007 0.007 156
Sad 0.964 ± 0.011 0.964 ± 0.009 0.964 ± 0.008 0.008 137
Best 0.969 ± 0.006 0.977 ± 0.008 0.973 ± 0.005 0.005 143
Glad 0.967 ± 0.013 0.936 ± 0.018 0.951 ± 0.012 0.012 149
Scared 0.966 ± 0.009 0.946 ± 0.015 0.956 ± 0.010 0.010 134
Stiff 0.957 ± 0.014 0.978 ± 0.007 0.967 ± 0.008 0.008 162
Surprise 0.944 ± 0.016 0.978 ± 0.009 0.961 ± 0.011 0.011 138
Table 5. Model Complexity and Efficiency Metrics.
Table 5. Model Complexity and Efficiency Metrics.
Model Parameter Training Time/Epoch (s) Inference Time (ms) Memory Usage (MB)
LSTM 2.3M 12.4 15.2 245
BiLSTM 4.1M 18.7 23.8 389
iBiLSTM 6.8M 25.3 31.4 512
Table 6. Evaluation matrices for all 7 classes.
Table 6. Evaluation matrices for all 7 classes.
Class Precision Recall F1-Score
Bad 0.974194 0.967949 0.971061
Sad 0.963504 0.963504 0.963504
Best 0.969466 0.976923 0.97318
Glad 0.966887 0.935897 0.95114
Scared 0.965517 0.945946 0.955631
Stiff 0.956522 0.977778 0.967033
Surprise 0.944056 0.978261 0.960854
Table 7. Comparative Validation Accuracy of LSTM-Based Classifiers.
Table 7. Comparative Validation Accuracy of LSTM-Based Classifiers.
Classifiers Validation Accuracy
LSTM 86.45%
BiLSTM 89.34%
iBiLSTM 95.71%
Table 8. Dataset-wise Performance of the Proposed Framework.
Table 8. Dataset-wise Performance of the Proposed Framework.
Dataset Sign Language No. of Classes Accuracy (%) Precision Recall F1-Score
PkSLMNM PSL 7 95.79 0.958 0.956 0.957
RWTH-PHOENIX-2014T GSL Continuous 93.84 0.936 0.934 0.935
SILFA ASL Dynamic 94.21 0.942 0.941 0.941
Table 9. Cross-Dataset Comparison with Existing Methods.
Table 9. Cross-Dataset Comparison with Existing Methods.
Study Model Dataset(s) Manual + Non-Manual Accuracy (%)
[13] SVM + CRF RWTH-BOSTON ✓ 84.0
[17] CNN-LSTM MArSL ✓ 72.4
[18] C3D + LSTM PkSLMNM Partial 87.35
[19] Transformer PHOENIX14 Manual only 82.66
Proposed Method YOLOv8 + VGG16 + C3D + iBiLSTM PkSLMNM, RWTH, SILFA ✓✓ 95.79
Table 10. Ablation-style Comparison Across Modalities.
Table 10. Ablation-style Comparison Across Modalities.
Feature Configuration Accuracy (%)
Facial Features only (YOLOv8 + VGG16 + RNN) 88.42
Body Gestures only (Improved C3D) 90.17
Fusion + LSTM 92.36
Fusion + BiLSTM 94.08
Fusion + iBiLSTM (Proposed) 95.79
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.