Intelligent Sign Language Interpretation System Using Multi-Modal Deep Learning Architectures

1. Introduction

Sign language is a important medium of communication for millions of people worldwide, providing a lifeline for those who are deaf or hard of hearing. Unlike spoken languages, sign language uses hand movements, facial expressions, and body postures to convey meaning, allowing users to express themselves fully without relying on sound. American Sign Language (ASL), for instance, stands as one of the most widely recognized sign languages and is the fourth most commonly used language in the United States. The growing prevalence of hearing loss—an issue affecting over 466 million people globally—underscores the increased need and importance of sign language for daily interactions and social connectivity. In the United States alone, estimates suggest that between half a million and two million individuals use ASL as their primary language, highlighting its critical role within both the deaf community and beyond. What sets sign language apart is its classification as a complete and natural language, possessing its own grammar, syntax, and linguistic structure, much like any spoken language. This realization came into sharper focus in the 1960s thanks to the groundbreaking work of William Stokoe, whose research at Gallaudet University revealed the advanced linguistic properties of sign languages. This shift in academic perspective led not only to the recognition of sign languages‘ sophistication but also to their cultural significance for millions of users. Today, colleges and universities across the United States are seeing record growth in ASL enrollment, with more than a hundred institutions accepting ASL to fulfill foreign language requirements, a trend that mirrors the increasing acceptance and legitimacy of sign language as an academic discipline.

Despite its rise, sign language users frequently encounter barriers in communication with the hearing community, largely due to the limited availability of qualified interpreters. Interpreters are often expensive, with costs $50 to $150 per hour, making their services inaccessible in many everyday or urgent situations. This shortage is especially critical in contexts like healthcare, legal proceedings, and emergencies, where clear communication can be a matter of safety and well-being. Furthermore, approximately 80% of deaf individuals worldwide struggle with written language, largely because of reduced educational opportunities, making sign language indispensable for their communication, education, and personal development.

The application of sign language extends beyond those who are deaf or hard of hearing—serving as a powerful communication alternative for people with autism spectrum disorders, Down syndrome, cerebral palsy, and other conditions that impact verbal communication. This broad utility highlights the universal principles of accessibility and inclusion that underpin the development of sign language and its wider adoption. Addressing these challenges, new technological solutions are increasingly sought to bridge the communication gap between signers and non-signers.

The vision of this project is to develop an AI-powered sign language interpretation system that delivers accurate, fast, and reliable visual-to-text or voice translation in real-world scenarios. Such technology aims to reduce dependence on human interpreters, offering a more accessible and affordable alternative for individuals who use sign language as their main mode of communication. Beyond technological advancement, the project seeks to strengthen social inclusion and promote communication equity—empowering deaf and hard-of-hearing individuals to participate more fully in society while fostering greater understanding and connection within the broader community.

2. Literature Review

Recent research in sign language recognition has increasingly focused on deep learning techniques to improve recognition accuracy and enable real-time interpretation. Kothadiya et al. [1] proposed a deep learning approach for sign language detection and recognition using Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks for temporal sequence modeling. The study focused on Indian Sign Language (ISL) and used a custom dataset containing 11 unique signs and video sequences. The reported training and testing accuracies were 97% and 94%, respectively. However, real-time performance was not evaluated.

Vyavahare et al. [10] developed a system for detecting and interpreting Indian Sign Language using LSTM networks for temporal sequence modeling.

Biravi and Krishnaveni [2] investigated sign language recognition using a Convolutional Neural Network (CNN)-based MobileNet architecture with transfer learning. The study focused on American Sign Language (ASL) and used the Kaggle ASL dataset with image preprocessing and data augmentation techniques. The reported training, validation, and testing accuracies were 90.22%, 88.5%, and 87.8%, respectively. The study demonstrates the applicability of transfer learning and lightweight CNN architectures for image-based sign language recognition.

Al-Qurishi et al. [3] presented a comprehensive survey of deep learning techniques for sign language recognition, covering CNN, Recurrent Neural Network (RNN), Hidden Markov Model (HMM), Support Vector Machine (SVM), Principal Component Analysis (PCA), and 3D CNN approaches. The study analyzed more than 50 datasets and compared the performance of different recognition techniques across multiple sign languages. The reported comparative accuracy ranged from approximately 65% to 95% across the different approaches. The survey highlighted the diversity of existing sign language recognition methods and the challenges associated with selecting appropriate architectures and datasets.

Baihan et al. [4] proposed a modified deep learning network and hybrid optimization approach for American Sign Language recognition. The architecture combined CNN-based spatial feature extraction with self-attention and LSTM-based temporal modeling. The approach also incorporated Mask R-CNN for background removal, VGG-16 for feature extraction, and a Hybrid Optimizer based on HOA and PFA. The study reported training, validation, and testing accuracies of 98.7%, 96.2%, and 95.8%, respectively, with a reported real-time processing time of approximately 60 ms. The work demonstrates the benefits of combining spatial and temporal feature extraction with optimization techniques.

Saiful et al. [5] proposed a real-time sign language detection system using CNN, MediaPipe, and OpenCV. The system processed camera input and used hand detection techniques to support real-time recognition. The reported accuracy, precision, recall, and F1-score were 98.6%, 99%, 99%, and 99%, respectively, with real-time processing at approximately 30 frames per second. However, the system was limited to 11 common gestures, restricting its vocabulary compared with broader ASL recognition systems.

Imran et al. [6] developed a real-time American Sign Language detection system using the YOLOv9 object detection framework. The work focused on multi-object detection and real-time sign recognition. Although the study demonstrated that YOLOv9 is capable of supporting real-time detection, the paper did not specify an overall accuracy value and required high-performance GPU resources. This indicates the potential of modern object detection architectures while also highlighting computational requirements as a consideration for practical deployment.

Ningsih et al. [11] applied the YOLOv5 algorithm to sign language detection to promote communication equality for people with disabilities.

Alsolai et al. [7] proposed an automated sign language detection and classification approach using a hybrid deep learning architecture combined with the Reptile Search Algorithm for optimization. The approach integrated MobileNet, CNN, and LSTM components along with MRFO and RSA evolutionary optimization strategies. The study reported training, validation, and testing accuracies of 99.51%, 97.8%, and 96.5%, respectively. The approach outperformed the reported SVM, Random Forest, and traditional CNN comparisons. However, the complexity of the optimization and hybrid architecture presents limitations for practical deployment.

Kumar et al. [8] investigated real-time sign language detection using a pre-trained VGG16 architecture with an attention mechanism. The attention mechanism was integrated with VGG16 to improve spatial feature representation. The study reported an accuracy of 97.5% using VGG16 alone and 99.8% after incorporating attention. The approach was designed for real-time processing but was limited to 23 gestures and required GPU resources, indicating a trade-off between recognition performance and computational requirements.

Verma et al. [9] proposed an approach for enhancing sign language detection using MediaPipe landmark detection and Convolutional Neural Networks. The system extracted 21 hand landmarks consisting of spatial coordinates and combined the landmark information with CNN-based classification. The study used an American Sign Language alphabet dataset containing alphabet and control classes such as space, delete, and nothing, with webcam-based data collection. The reported accuracy was 99.12%, with precision, recall, and F1-score of 98.8%, 98.9%, and 98.85%, respectively. This work demonstrates the usefulness of combining hand landmark information with visual features for sign language recognition.

The reviewed studies demonstrate the progression of sign language recognition from conventional deep learning models toward real-time, attention-based, landmark-based, and hybrid architectures. CNN-based methods provide effective visual feature extraction, while LSTM and GRU networks are useful for modeling temporal information. MediaPipe-based approaches provide structured hand landmark information, whereas object detection and attention mechanisms improve the capability of systems to identify relevant visual regions. However, several existing approaches remain limited by restricted gesture vocabularies, computational requirements, lack of real-time evaluation, or reliance on a single type of representation. These observations motivate the development of a multi-modal architecture capable of combining complementary visual and spatial information for more robust sign language recognition.

3. Methodology

The proposed sign language interpretation system follows a multi-modal deep learning architecture that combines visual and spatial information for sign recognition. The overall methodology consists of data preprocessing, feature extraction, multi-modal model processing, ensemble-based prediction, model training, and real-time camera-based recognition.

The input to the system is obtained from video frames or a live webcam feed. The frames are processed to focus on the signer’s hands and are resized to 224 × 224 pixels for processing by the CNN and Vision Transformer (ViT) models. The pixel values are normalized to maintain consistent input ranges. Data augmentation techniques such as rotation and flipping are applied to improve model robustness. In addition, MediaPipe is used for hand landmark detection, where key hand points are extracted and provided as input to the Graph Neural Network (GNN). Background removal can also be applied to reduce unwanted visual information.

The system employs three complementary deep learning components. A CNN-based model, using ResNet architecture, processes the image frames and extracts hierarchical visual features representing hand shapes and movements. The GNN processes the detected hand landmarks, typically consisting of 21 keypoints with their spatial coordinates, and learns the relationships between joints and fingers. The Vision Transformer processes the resized images by dividing them into image patches and applying self-attention to identify important spatial information.

The outputs of the CNN, GNN, and ViT models are combined using an ensemble learning strategy. Weighted averaging or confidence-based fusion is used to combine the predictions from the individual models. This multi-modal fusion helps compensate for the limitations of individual models and improves the stability of sign recognition under varying conditions such as changes in lighting and background.

For model training, a batch size of 32 samples is used with up to 200 training epochs and early stopping to reduce overfitting. The Adam optimizer is used with a learning rate of 0.001, while categorical cross-entropy is used as the loss function for multi-class classification. The dataset is divided into training, validation, and testing sets using an 80%, 10%, and 10% split, respectively. Model performance is evaluated using accuracy, precision, recall, F1-score, word error rate (WER), and confusion matrix analysis.

For real-time recognition, webcam frames are continuously captured and processed by the system. The relevant region of interest is extracted and passed through the preprocessing and multi-modal inference pipeline. The predictions generated by the CNN, GNN, and ViT models are fused to produce the final sign prediction along with its confidence score. The resulting prediction is then displayed through the real-time application interface. This workflow enables the system to perform sign language recognition using both visual and hand-landmark information while supporting practical real-time operation.

4. Results

The developed sign language interpretation system was evaluated on the ASL Alphabet dataset using an 80/10/10 train/validation/test split. The architectures tested were a custom Convolutional Neural Network (CNN), ResNet-18 with transfer learning, Vision Transformer (ViT), Graph Neural Network (GNN), and a multi-modal ensemble of CNN, GNN, and ViT. The comparative experimental results show that the custom CNN achieved an accuracy of 99.83%, the Vision Transformer achieved 93.96%, and the GNN achieved 83.0%, while ResNet-18 achieved 90.0%. The proposed ensemble approach combining CNN, GNN, and ViT achieved an accuracy above 95%, demonstrating the potential of multi-modal model integration for sign language recognition.

The detailed evaluation further provided class-wise performance analysis using precision, recall, F1-score, and confusion matrices. The Vision Transformer achieved an accuracy of 93.96%, with a precision of 94.18%, recall of 93.96%, and F1-score of 0.9396. The macro-average and weighted-average F1-scores were both approximately 0.94. The class-wise results indicate strong recognition performance across most sign classes, although variations were observed for visually similar signs.

The detailed CNN evaluation reported an accuracy of 99.83%, with precision, recall, and F1-score of approximately 0.9983. The corresponding macro-average and weighted-average F1-scores were approximately 1.00. The confusion matrix showed that the majority of predictions were concentrated along the diagonal, indicating a high level of correct classification across the evaluated sign classes.

The GNN achieved an accuracy of 83.0%, with a precision of 84.0%, recall of 82.5%, and F1-score of 0.825. The class-wise evaluation demonstrated that the GNN was effective for several classes but showed comparatively lower performance for certain signs, particularly those requiring more precise structural discrimination.

In addition to classification performance, the system was implemented as a real-time web-based application. The deployment architecture uses a Flask and SocketIO backend, an HTML5 and JavaScript frontend, WebSocket-based communication, region-of-interest extraction, and server-side model inference. The system accepts a webcam stream at 640 × 480 resolution and approximately 30 frames per second. A 200 × 200 region of interest is extracted from the input, followed by model inference using a normalized 224 × 224 input. The predicted sign and confidence score are then displayed through the real-time interface.

The system also provides real-time confidence visualization, FPS monitoring, progressive word construction, auto-complete suggestions, sentence construction using the SPACE gesture, and deletion using the DELETE gesture. These features demonstrate that the developed system extends beyond static classification toward practical real-time sign language interpretation.

5. Discussion

The experimental results demonstrate that different deep learning architectures provide different advantages for sign language recognition. The CNN showed strong performance in visual pattern recognition, indicating that convolution-based feature extraction is effective for identifying spatial characteristics of static hand signs. Its comparatively lower computational requirements also make CNN-based approaches suitable for practical deployment.

The Vision Transformer also demonstrated strong recognition capability, achieving approximately 94% accuracy in the detailed evaluation. Its attention-based architecture enables effective learning of spatial relationships within the input image. However, the higher computational requirements associated with transformer-based models remain an important consideration for real-time deployment.

The GNN achieved comparatively lower overall performance than the CNN and ViT. Although graph-based learning provides a useful representation of structural relationships between hand landmarks, its performance depends strongly on the quality of landmark detection and preprocessing. The class-wise results show that some signs were considerably more difficult for the GNN to distinguish, indicating limitations in relying primarily on landmark-based structural information.

The comparative results support the use of a multi-modal approach. The ensemble combining CNN, GNN, and ViT achieved an accuracy greater than 95% in the reported experimental results. The combination of visual features, structural relationships, and attention-based representations provides architectural diversity that can improve robustness compared with relying on a single model. This suggests that combining complementary representations is a promising approach for sign language interpretation.

The real-time implementation further demonstrates the practical applicability of the proposed system. The integration of webcam input, region-of-interest processing, model inference, confidence estimation, and temporal finger-spelling logic enables the system to generate predictions continuously. The addition of word construction, auto-complete, sentence formation, and deletion functions improves the usability of the system beyond simple letter classification.

Despite the strong recognition performance, the current system has limitations. The system primarily focuses on individual sign recognition and therefore does not yet provide complete word- or sentence-level sign language understanding. The computational requirements of some architectures may also affect deployment on resource-constrained devices. Furthermore, accurate interpretation of complete sign language communication may require additional information such as facial expressions, body posture, and temporal movement patterns.

6. Conclusions

This work presented a multi-modal deep learning based system for real-time sign language interpretation. The proposed approach investigated multiple architectures, including CNN, ResNet-18, Vision Transformer, and Graph Neural Network, and explored their integration through an ensemble framework. The experimental results demonstrated strong recognition performance, with the reported ensemble achieving greater than 95% accuracy.

The detailed evaluation further demonstrated the effectiveness of the individual architectures, with the CNN achieving 99.83% accuracy in the detailed evaluation, the Vision Transformer achieving 93.96%, and the GNN achieving 83%. These results indicate that convolutional, attention-based, and graph-based architectures capture complementary characteristics of sign language data.

The developed system was also integrated into a real-time web-based application capable of processing webcam input and displaying predicted signs with confidence scores. Temporal processing and finger-spelling functions enable recognized letters to be progressively converted into words and sentences. Overall, the results demonstrate the feasibility of combining deep learning models with real-time processing to develop an accessible sign language interpretation system.

Future improvements can focus on extending recognition from individual letters to complete words and sentences, incorporating facial and body information, improving computational efficiency for mobile deployment, and supporting two-way communication between sign language and spoken or written language.