Approach to Real-Time Sign Language Recognition based on Data Fusion with Metagloves and Orbbec Depth Camera

1. Introduction

Communication is fundamental to our lives as human beings. However, for those who have impaired hearing and speaking capabilities, proper communication is a big problem. Though there is a lot of usage of sign language among deaf and mute individuals[1], people from the rest of the world do not understand sign language. Thus, there is a clear gap in communication, which can only be resolved through the help of appropriate assistive technologies that can facilitate better interaction between people[2]. Recently, gesture recognition has been a field of interest among researchers in the domain of human computer interaction. The existing systems for recognizing signs are broadly classified into two types – one is vision-based and the other one is based on sensors[3]. In vision-based systems, the use of cameras results in problems in recognition due to problems such as bad lighting conditions, background noise, etc. On the other hand, sensors such as data gloves can provide the correct data about the fingers’ movement, but cannot provide any position information about the hand[4].To address these issues, this study suggests a hybrid technique in which Manus MetaGloves have been paired with Orbbec depth camera sensor. The MetaGloves sensors can be employed for capturing hand gestures and fingers’ movement, while Orbbec device captures the spatial information related to hand including position, orientation and distance[5]. Thus, the proposed system becomes highly efficient and accurate in terms of recognizing the gestures.This captured information will then be used in order to accurately recognize the signs of sign languages and convert these signs into texts and spoken sounds. It will allow disabled people who lack hearing capabilities as well as the ability to speak to communicate freely and comfortably with other people. Moreover, the usage of Unity framework facilitates the interaction process.This project will primarily focus on development of efficient sign language recognition system using Manus MetaGloves and Orbbec sensors in conjunction with the Unity platform[6].

Figure 1 shows the suggested system combines the Manus MetaGloves and Orbbec Depth Camera in order to perform precise recognition of sign language in real-time. Manus MetaGloves provide accurate data on finger joint angles, hand orientation, and movement, giving detailed information on the finger movements. In parallel with that, the Orbbec Depth Camera receives three-dimensional hand position coordinates (X, Y, Z), hand orientation, and depth information, which makes it possible to track the hand gestures spatially. Data from sensors are synchronized in the process of data acquisition and then preprocessed in the preprocessing module by removing noise, smoothing the data, and segmenting the hand region. The obtained data are calibrated and aligned to ensure that there is a common coordinate system for both the gloves and the depth camera. After calibration, the data are passed to the data fusion module, where finger motion information from the MetaGloves is combined with the spatial hand position information received by the depth camera, providing a complete set of hand gesture information. The data are further processed by the gesture recognition module to recognize sign language gestures by rule-based algorithms or machine learning techniques. To perform real-time monitoring and visualization, the fused hand data is presented in the form of a 3D hand model in the Unity platform. In the end, the recognized gestures are transformed to meaningful outputs using the output generation component, providing text and speech output. This provides the ability for communication between sign language users and non-sign language users.

Several research papers have already been done regarding Hand Gesture Recognition (HGR). In order to have an adequate review, recent advances in HGR system can be classified according to vision-based approach, sensor-based approach and sensor fusion.

2. Vision-Based Approaches

Vision-based HGR system uses RGB or depth camera in order to capture hand gestures and does not require any wearable devices by the user. Vision-based HGR system has made many developments due to application of the deep learning technique. Recent research papers have used CNN (Convolutional Neural Networks), Vision Transformers and multimodal fusion model which uses RGB-D dataset in order to enhance recognition rate reaching accuracy of 97.28% [8].

There are several limitations that vision-based HGR system has. Accuracy of recognition is very sensitive to occlusion when the hand/fingers are partially or completely obstructed from the view of the camera. Variable illumination conditions like low-light environments, over-exposed images and changing light conditions affect vision-based HGR system performance negatively [9]. Moreover, there are background noise and clutter in the real environment that interfere the process of hand segmentation and feature extraction. Recent reviews which are performed between 2018-2024 prove that environmental sensitivity is the main problem of vision-based HGR systems.

2.1. Sensor-Based Approaches

In sensor-based approach, sensors like smart glove having flex sensors, IMUs, and sEMG sensors are used to track hand kinematics. It allows very precise measurement of finger joint motion and orientation of hand independent of the environment lighting and visual occlusion.

Some recent works which use smart data glove along with deep learning techniques have shown excellent results. For instance, in one study Fan et al. (2023) have designed a smart data glove which is suitable for amphibious environment having recognition accuracy of 98.21% [10]. In another work by Wu et al. (2023), a CNN-BiLSTM model along with data glove inputs has been used giving an accuracy of 95.05% and precision of 95.43% [11].

Although these techniques have shown very precise tracking capability of fine-grained finger motion, yet the major drawback of this technique is that it lacks absolute spatial positioning information, therefore making it impossible to detect gestures based on the hand location in 3D space.

2.2. Sensor Fusion Approaches

To counteract the weaknesses of single-modality systems, there has been increasing emphasis on the use of sensor fusion algorithms which exploit multiple complementary data sources. Through the integration of the high precision kinematics provided by smart gloves and spatial sensing via depth cameras, hybrid solutions are able to provide higher robustness and accuracy.

There have been several attempts at using sensor fusion reported recently in the literature. Hashi et al. (2024) found that by combining sEMG sensors with depth camera information from the Leap Motion controller, recognition accuracy could be increased from 86% to 96% [12]. Multi-modal fusion algorithms have been developed which use heterogeneous networks for feature extraction and fusion from various sensors.

The table below summarizes the key related works and their reported accuracies:

Study	Approach	Modality	Accuracy	Limitations that were solved
Hao et al. (2024) [8]	Fusion of multiple features	Vision (RGB-D)	97.28%	Occlusion and complex situations
Wu et al. (2023) [11]	CNN-BiLSTM with glove	Sensor-only	95.05%	Lighting independence
Fan et al. (2023) [10]	Smart data glove	Sensor-only	98.21%	Cross user identification
Hashi et al. (2024) [12]	sEMG + depth camera	Sensor fusion	86-96%	Complementary multimodality
Colli Alfaro et al. (2022)	Fusion of EMG and IMU	Sensor fusion	94%	User independent classification
Proposed system	MetaGloves with Orbbec	Sensor + Vision Fusion	86.35%	Occlusion, lighting and spatial

Our proposed method takes advantage of all these contributions through a unique approach of integrating the millimeter accuracy in tracking the finger movement by means of the Manus MetaGloves (with drift-free and occlusion-insensitive kinematic information) with the ability of the Orbbec camera to sense the depth information of the hand in real time (absolute position information). The proposed method compensates for the weaknesses in both vision-based and sensor-based approaches at the same time.

3. Methodology

This paper aims at designing a practical and a hybrid model for sign language recognition that combines using a combination of Manus MetaGloves and Orbbec depth sensing camera. Instead of employing one type of input data, this model follows a hybrid approach where two different sources of input data will be utilized simultaneously – motion input and vision input. The whole process is divided into several stages for optimal precision[7].Data collection in this model is performed through use of two different devices. First, Manus MetaGloves will be employed for capturing hand and finger motion information. This device captures the exact angle of fingers bending as well as the hand orientation in general. Thus, this tool can provide the most accurate data on finger movement which is necessary for sign language recognition. Second, Orbbec depth sensing camera will be used for gathering three-dimensional information regarding hand position. In particular, it captures the X coordinate, Y coordinate, Z coordinate, as well as the distance of hand movement. As a result, the exact location of hands can be easily established by the system. Overall, two different types of data collection are needed for optimal efficiency[8].After collection of the raw data, this information cannot be used immediately since it might contain some unnecessary noise and variance. In order to make sure that there are no distortions and irregularities, the preprocessing stage takes place during which all irrelevant information gets excluded. The glove sensor readings are smoothened in order to avoid sudden jumps while only the hand area from the depth camera information is left[9].It should be noted that a major problem in the current setup is the presence of two devices working with data coming from various coordinate systems. However, in order to achieve proper functioning of the system, it is crucial to find ways for proper data processing and alignment. In other words, a calibration process takes place where both coordinate systems are aligned within a single frame of reference. Here, the transformation technique is employed allowing mapping the readings from one coordinate system to another. In other words, proper alignment of the finger gestures is achieved with the help of this technique[10].Data fusion follows the process of alignment. Data fusion is the heart of the entire system. During this process, the data captured by the MetaGloves in terms of finger movements is fused with that captured by the depth camera in terms of position in space[11]. Through this fusion of data, a full picture of the hand can be formed.This data consists of finger movement data, hand orientation data, and location data of the hand. The use of the two devices together enhances the effectiveness of the system compared to when only one is used since it increases its reliability and accuracy.The collected data is used to control a model of the hand in a 3D environment using Unity[12]. For every finger movement made with the glove, there is an equivalent action in the 3D hand model, and the positional data obtained from the camera is used to move the whole model accordingly, resulting in a real-time visualization of the hand on the screen.Using Unity in this process gives users a better interface through increased interactivity and proper visualization of gestures.After obtaining a 3D model of the hand, the next step involves recognizing various gestures that the user makes. This is achieved through analysis of the collected data and comparison of that data against predefined gestures. The identification of simple gestures involves the implementation of rule-based recognition systems.Some of the common rules include closing a fist when all fingers bend above a certain threshold point and opening up the hand if all fingers are extended[13]. Machine learning algorithms can also be employed to achieve advanced results.This step involves translating the recognized gesture into output that is understandable by other people. The gesture is translated either into text or speech depending on the preference. The output generation is a process that takes place in real-time[14].It is apparent that the entire system will work effectively in terms of accuracy, efficiency, and convenience. This is because the proposed methodology incorporates both hand gesture recognition using the glove and depth estimation.[15]

The main components used are Manus MetaGloves along with an Orbbec depth sensor camera to enable proper detection of hand gestures[16].Manus MetaGloves are advanced wearables that offer real-time motion tracking of hand gestures and finger movements[17]. Multiple sensors embedded into the gloves capture all the movements and track the orientation of hands. Finger bending along with joint angle measurement enables accurate tracking of finger movement. The gloves send real-time motion data using the dedicated software application interface which includes finger flexion values and orientation of hand gestures. A major advantage offered by MetaGloves is that they provide accurate tracking of gestures even in changing environmental conditions like varying light[18].

The gesture recognition system using the smart glove obtains finger movement and hand orientation through flex sensors, IMU sensors, and pressure sensors[19]. Data obtained from the sensors is then sent wirelessly using the Bluetooth protocol to the computer, where it is processed and shown on the computer[20]. Signal filtering, joint angle computation, and hand orientation calculation are done for accurate gesture recognition. The data processed is used for 3D visualization of the hand, gesture recognition, and production of text and speech output[21]. Orbbec depth sensing camera is used to record three-dimensional data on hand spatial location[22]. The depth camera differs from the typical RGB camera since the latter records the distance between the sensor and the object, thus providing a 3D tracking capability.The camera records such factors as X, Y, and Z coordinates, orientation, and hand movement[23]. These factors are extremely important in determining the position of the hand in relation to the user and the surroundings. Depth sensing also eliminates some of the limitations of regular vision-based methods, such as being sensitive to lighting and background[24]. The Orbbec Depth Camera uses the RGB, infrared, and depth sensors to capture depth information in real-time. The data obtained from the sensors are used to create a depth map that defines the distances between objects in 3D space[25]. The system then proceeds to conduct skeleton and hand tracking to identify hand position and joint coordinates (X, Y, Z)[26]. After the data have been processed to filter noise, depth information, point clouds and features are extracted. Finally, the data are transferred to the host computer through the USB interface[27].

3.1. Dataset Collection and Description

The data collection process has been done in the laboratory setting in the Department of Biomedical Engineering at the Easwari Engineering College in Chennai, India. The dataset has been collected using the Manus MetaGloves and Orbbec Depth Camera operating together so that both types of data are acquired for each gesture.

3.2. Devices Used for Data Collection

Following physical devices were used for data collection:

Figure 2. depicts the experimental setup that was used for gesture acquisition in sign language. The participant dons a MANUS MetaGlove that uses motion sensors to acquire hand and finger kinematics. Joint position, orientation, and finger articulation are streamed to the host computer via the MANUS Core SDK. An Orbbec depth sensing camera is set up in front of the participant to acquire hand gestures along with their depth. The data obtained from both sensors are then processed on the workstation that uses MANUS software. There, the real-time skeletal hand model is constructed and visualized. On the laptop interface, the skeleton hand can be viewed to continuously monitor the hand pose estimation and sensor calibration.

The synchronized glove and vision data are then used to build the gesture dataset that includes spatial, temporal, and kinematic features. Multimodal dataset is then used for gesture segmentation, feature extraction, and sign language recognition using deep learning algorithms. Combination of motion sensing with vision helps to increase gesture acquisition accuracy and eliminate ambiguities that may occur due to occlusions and finger articulations.

Figure 2.1. Hand gestures from MANUS Metagloves.

Figure 2.1. Hand gestures from MANUS Metagloves.

Manus MetaGloves: The Manus MetaGloves are sophisticated wearables that allow for real-time hand gesture and finger tracking. They have EMF (Electro Magnetic Field) receivers in the fingertips that constantly capture the EMF created by a central transmitter mounted on the top of the casing on the back of the hand. When fingers make movements in space, the receivers are able to pick up the strength and direction of the EMF that allows sophisticated algorithms to transform it into real-time 3D positioning and rotation data. It provides millimeter-precise fingertip tracking that is drift-free and not dependent on any occlusions. The gloves are available in various sizes, have swappable batteries, and require short calibration. The data is then transferred wirelessly through Bluetooth to the MANUS Core software platform on the host computer.

Figure 2.2. Hand depth Information from Orbbec depth sensing camera.

Figure 2.2. Hand depth Information from Orbbec depth sensing camera.

Orbbec Depth Sensing Camera: The Orbbec depth sensing camera is used for capturing three dimensional data of hand spatial positioning. The Orbbec depth sensing camera makes use of the Structured Light or Time-of-Flight (ToF) technology to capture depth information. The camera captures X, Y and Z coordinates, orientation, hand motion data and all this in real time at a rate of 30 FPS. The depth camera is not like the normal RGB camera in that it captures the distance between the sensor and the object. The depth camera therefore has the ability to provide 3D tracking.

Gesture Classification	Open Hand
Open/Fully Extended Gestures	V Sign, Pointing Gesture, Rock Sign
Finger Gestures	Closed/Pinch Gesture, Grasping Gesture
Close/Flexed Gestures	OK Gesture, Open Hand Form
Circular/Closed Gestures	Open Hand

Each gesture was captured in a number of instances to maintain the statistical reliability of the results. For each gesture, both kinematic parameters (MCP, PIP and DIP joint angles of all five fingers - thumb, index, middle, ring and pinky, acquired using MetaGloves) and spatial parameters (X,Y,Z coordinates and depth values, captured using Orbbec camera) were measured at a speed of 30 FPS.

3.3. Dataset Characteristics

In terms of spatial positioning and finger joint configuration, the following variations have been captured by the dataset:

• The depth values recorded varied from around 569 mm (for Grasping Gesture) to 4280 mm (for Victory Sign).
• This indicates large variation in the distance of hand from the camera in different hand gestures.
• The X and Y coordinates of the spatial positions were highly variable among different gestures.
• Angle at which the joints were configured was also different for the open hand and closed hand gestures, where the open hand gestures had small joint angles while the others had high joint angles in their middle, ring, and pinky fingers.

It is the combination of all the above data streams that forms the basis of the feature-level fusion method.

5. Practical Applications of the System

(b) Virtual Reality and Metaverse Interactions: The high-fidelity finger tracking provided by the Manus MetaGloves enables precise hand manipulation in virtual environments. Combined with the 3D spatial awareness from the Orbbec camera, the system can be used for VR gaming, virtual meetings, architectural walkthroughs, and collaborative virtual design sessions where natural hand gestures serve as the primary input modality. The Unity-based visualization framework facilitates direct integration with existing VR/AR applications.

(c) Medical Rehabilitation and Physiotherapy: The system can be employed in physical therapy settings to track and quantify the progress of patients recovering from hand injuries, carpal tunnel syndrome, or stroke-related motor impairments. The precise joint angle measurements (MCP, PIP, DIP) combined with spatial position data allow therapists to objectively measure the range of motion and recovery progress over time, providing real-time visual feedback to patients through the Unity 3D interface.

(c) Physical Therapy & Rehabilitation: This system can be used in physical therapy clinics in order to track the progress of hand rehabilitation in patients suffering from injuries, carpal tunnel syndrome, or neurological disorders after a stroke. The detailed tracking of joint angles (MCP, PIP, DIP) and spatial position would allow for an objective measurement of recovery and visual feedback for patients via Unity 3D visualization interface.

(d) Remote Education and Learning Sign Languages: This model may also be employed as an education tool to teach sign languages as it helps in giving instant feedback to the learner on whether his/her hand placements and finger placements are right. This is achieved through Unity’s 3D visualization tool where the learner compares their gestures with those of the reference gesture in real time.

5.1. Recognition Accuracy in Various Occlusions

One of the important features of the proposed system lies in its ability to resist occlusion, which happens to be one of the major difficulties of gesture recognition systems. In contrast with the pure vision systems, where occlusion results in a very low level of accuracy, the combination of kinematic information and depth information ensures high recognition performance in all occlusions.

Below is shown the recognition accuracy in different occlusions:

Occlusion Experiment	Description	Accuracy of Camera Only	Accuracy With Only Gloves	Accuracy of Fused System
No Occlusion	Full visibility of hand	80.45%	82.10%	86.35%
Partial Visual Occlusion (50%)	Hand partly obscured by object	Approximately 65%	82.10%	~84%
Total Visual Occlusion	Hand fully out of sight of camera	5% to 10%	82.10%	~82%
Finger to Finger Occlusion	Fingers overlapping in intricate movements	Approximately 70%	82.10%	~85%
Self Occlusion	Back of hand visible to camera	Approximately 55% to 60%	82.10%	~83%

As for the condition of complete visual occlusion, the Manus MetaGloves keep detecting finger gestures with the same level of precision because EMF tracking does not require any line-of-sight visibility. In this regard, the system uses only glove recognition, which guarantees around 82% of accuracy. As for partial occlusion situations, the fusion algorithm automatically adjusts the weight of glove recognition in favor of its use when depth detection through the camera is inaccurate.

5.2. Effect of Light Conditions on Recognition Results

The influence of light conditions on the process of gesture recognition is an important parameter which distinguishes the suggested fusion method from vision-based methods. Below there is an explanation of how light conditions influence the result:

Lighting Condition	Description	Accuracy of Camera Alone	Accuracy with only gloves	Accuracy of Fused System
Ideal Indoor Lighting	Normal lab lighting	80.45%	82.10%	86.35%
Low Light (Dark Room)	Dim lighting	Approx. 60-65%	82.10%	~84%
Bright Sunlight	IR radiation interference	Approx. 55-60%	82.10%	~83%
Frequently Changing Light	Variable lighting	Approx. 65-70%	82.10%	~84%
No Light (Absolute Darkness)	Zero ambient lighting	Approx. 0-5%	82.10%	~82%

Explanation of Results:

The conventional camera based gesture recognition system has an inherent dependence on lighting conditions. In situations where there is less than adequate lighting, the camera will not be able to capture enough detail, resulting in a poor recognition performance. When the lighting condition becomes overexposed or has glare, then the image contrast becomes lower and shadows are lost and it becomes hard to separate out the finger and hand boundaries.

The Orbbec depth camera uses IR structured light and Time of Flight (ToF) technologies that make it intrinsically insensitive to visible light when compared with the RGB camera. The illumination is done using IR that is supplied by the camera itself and provides reliable depth information irrespective of the lighting conditions in the room. However, the Orbbec camera may still be affected by bright sunlight (IR component is present in bright sunlight), reflective surfaces (IR light will scatter off reflective surfaces) and other IR sources.

Manus MetaGloves employ Electro Magnetic Field (EMF) tracking technology, which does not depend on ambient lighting at all. The EMF sensors are used to track position and orientation by detecting stable magnetic fields, not images. Therefore, no matter if the surrounding space is completely dark or brightly illuminated or the lighting condition changes very quickly, the finger tracking accuracy will always be the same.Since the suggested approach combines the information provided by both modalities, the recognition results become immune to lighting conditions. If the lighting becomes a factor, decreasing the accuracy of the depth camera (for example, direct sunlight), the suggested system switches automatically to using kinematic information from the glove, which is not influenced by lighting in any way.

6. Conclusion

In this work, a real-time sign language recognition system based on the sensor fusion technique with the Manus MetaGloves and the Orbbec Depth Camera has been described. The proposed technique involves the combination of finger motions detection from the gloves and hand tracking from the depth camera to avoid the disadvantages associated with the usage of the separate sensors. While the MetaGloves are responsible for fingers’ joints and hand posture recognition, the Orbbec Depth Camera detects hand positions and depth data. Experimental results show that the proposed sensor fusion technique allows achieving a recognition rate of about 86.35%, offering high reliability for translation of sign language gestures into texts and speech with minimum delays. In addition, the proposed technique helps increase the gesture recognition accuracy, avoid ambiguities due to occlusions and serves as an effective tool of communication between hearing impaired people and users without knowledge of sign languages. In the future, the work will be concentrated on extending the repertoire of gestures to encompass a broader range of dynamic and continuous sign language gestures. Some of the ways to improve the current system involve incorporating recognition models using deep learning techniques, increasing resistance to different lighting and complex background settings, and computational efficiency in order to operate in portable and embedded computing devices. Furthermore, future research will explore multilingual sign language translation, cloud-based communication.