SignFuse: A Proposed Dual-Stream Cross-Modal Framework for Gloss-Free Sign Language Translation with Large Language Models

1. Introduction

Sign language is the primary mode of communication for over 70 million Deaf individuals worldwide [1]. Unlike spoken languages, sign languages are visual-gestural systems that convey meaning through a rich combination of manual articulations (hand shapes, orientations, and movements), non-manual markers (facial expressions, mouthing, and eye gaze), and body posture [2]. Despite the linguistic complexity and cultural significance of sign languages, the communication barrier between sign language users and the hearing majority remains a significant societal challenge.

The rapid evolution of artificial intelligence from early symbolic systems to modern deep learning paradigms [3] has enabled transformative progress across numerous domains, including Sign Language Translation (SLT), which aims to automatically translate sign language videos into spoken language text. SLT is fundamentally more challenging than Sign Language Recognition (SLR), as it requires not only identifying individual signs but also understanding grammatical structures, handling word reordering between sign and spoken languages, and generating fluent natural language output [4].

Traditional SLT approaches rely on an intermediate gloss representation a written annotation of individual signsas a bridge between the visual and linguistic domains [2,4]. However, gloss annotations suffer from several critical limitations: (i) they are expensive and time-consuming to produce, requiring trained linguistic annotators; (ii) they are linguistically imprecise, as different sign languages have distinct grammatical structures from their spoken counterparts; and (iii) they fail to capture the full complexity of sign language, including non-manual features and simultaneous multi-channel information. These limitations have motivated a growing body of research on gloss-free SLT, which directly translates sign language videos into spoken language without intermediate glosses [5,6].

The recent emergence of Large Language Models (LLMs) [7,8] and, more broadly, Multimodal Large Language Models (MLLMs)as comprehensively surveyed in [9]has catalyzed significant progress in gloss-free SLT. Models such as SignLLM [10], Sign2GPT [11], and SpaMo [12] have demonstrated that the powerful language generation capabilities of LLMs can be effectively harnessed for sign language translation when provided with appropriate visual representations. More recently, MMSLT [13] leverages multimodal LLMs to generate textual descriptions of sign language components, while SCOPE [14] introduces context-aware SLT for dialogue scenarios.

Despite these advances, we identify two critical limitations in existing approaches that motivate our work:

Limitation 1: Underutilization of complementary modalities. Current LLM-based SLT methods predominantly rely on a single visual modalityeither RGB frames processed through CNNs or Vision Transformers [11,12], or pose-based skeletal data [15]. Sign language, however, is inherently multimodal, encoding information through multiple simultaneous channels. Appearance-based features from RGB frames capture hand shapes, finger configurations, and facial expressions, while skeletal features capture joint positions, movement trajectories, and limb articulations. These two modalities provide complementary information that is essential for accurate translation. No existing LLM-based SLT approach effectively fuses these complementary modalities at a deep feature level through learned cross-modal interactions.

Limitation 2: Insufficient temporal modeling. Sign language operates at multiple temporal granularities: individual hand configurations are captured within single frames, signs typically span 5–30 frames, and sentences require understanding long-range dependencies across hundreds of frames. Existing approaches typically use a single temporal resolution through fixed-stride feature sampling or simple temporal attention, which inadequately captures the hierarchical temporal dynamics of sign language. This limitation is particularly acute for translating compound sentences, temporal modifiers, and discourse-level structures.

To address these limitations, we propose SignFuse (Sign Language Fused Translation), a novel theoretical and architectural framework that introduces three key innovations:

• Dual-Stream Feature Extraction: We employ a CNN-based visual stream (ResNet-50) and a GCN-based skeletal stream (ST-GCN) to extract complementary features. The visual stream captures appearance information (e.g., hand shapes, facial expressions), while the skeletal stream captures structural and kinematic information (e.g., joint angles, movement trajectories).
• Cross-Modal Fusion Attention (CMFA): We introduce a novel bidirectional cross-attention module that enables each modality to attend to the other, producing enriched multimodal representations that leverage the strengths of both visual and skeletal features through learned gated aggregation.
• Hierarchical Temporal Aggregation (HTA): We design a multi-scale temporal modeling module that captures sign language dynamics at frame, segment, and sequence levels, enabling the model to understand both fine-grained gestures and long-range contextual dependencies through adaptive temporal fusion.

Furthermore, we propose a progressive multi-stage training strategy that systematically: (i) pre-aligns visual and skeletal features via contrastive learning, (ii) aligns fused multimodal features to the LLM’s input space, and (iii) performs end-to-end fine-tuning with LoRA [16] for efficient adaptation. An overview of the proposed framework is shown in Figure 1.

Because training end-to-end MLLMs on massive video datasets requires substantial GPU compute clusters (typically 4–8× A100 GPUs for days of continuous training), empirical validation remains a barrier for many fundamental researchers. Therefore, this paper is presented as a formal architectural blueprint and research hypothesis. We provide the complete mathematical formulation, detailed architectural specifications, and a fully implemented PyTorch codebase. We outline the complete structural design of SignFuse and propose a rigorous experimental protocol for future evaluation. We open-source our theoretical framework to the community, inviting institutions with the necessary computational resources to execute the training phases and collaborate with us on validating this highly promising paradigm.

Contributions. Our contributions are summarized as follows:

• We propose SignFuse, a novel dual-stream architecture that, for the first time, integrates CNN-based visual features and GCN-based skeletal features through deep bidirectional cross-modal attention for LLM-based sign language translation.
• We introduce the Cross-Modal Fusion Attention (CMFA) module with gated aggregation and the Hierarchical Temporal Aggregation (HTA) mechanism, both of which are formally specified with complete mathematical formulations.
• We design a three-stage progressive training strategy that systematically bridges the gap between multimodal sign language representations and the linguistic space of LLMs.
• We provide a complete, runnable PyTorch implementation and formalize a comprehensive experimental protocol on three standard benchmarks, extending an open call for community-driven empirical validation.

2. Related Work

2.1. Sign Language Recognition

Sign Language Recognition (SLR) has progressed through several paradigm shifts driven by advances in computer vision and deep learning. Early approaches relied on handcrafted features such as HOG descriptors and color histograms, combined with Hidden Markov Models (HMMs) for temporal modeling [17]. The introduction of Convolutional Neural Networks (CNNs) brought significant improvements, with methods like Recurrent Convolutional Neural Networks [18] combining spatial feature extraction with temporal sequence modeling via LSTMs [19]. Three-dimensional CNNs [20] extended this approach by jointly modeling spatial and temporal dimensions.

The Transformer architecture [21] has increasingly dominated the SLR landscape, with works such as [4] and [22] demonstrating strong sequence modeling capabilities through self-attention mechanisms. More recently, ConSignformer [23] introduced a Transformer backbone specifically designed for continuous SLR, incorporating convolutional position encoding and efficient attention patterns optimized for video understanding.

Skeleton-Based Approaches. Graph Convolutional Networks (GCNs) have emerged as powerful tools for skeleton-based sign language recognition [24]. By naturally representing the body as a graph with joints as nodes and bones as edges, GCNs exploit the structural relationships between body parts. The Spatial-Temporal Graph Convolutional Network (ST-GCN) [25] extended this paradigm by incorporating temporal edges between corresponding joints across frames, enabling joint spatial-temporal reasoning. These approaches have shown particular promise for capturing the kinematic properties of signs, such as trajectory shapes and velocity profiles.

MediaPipe-Based Systems. Lightweight pose estimation frameworks like MediaPipe Holistic [26] have enabled real-time skeleton extraction, making pose-based approaches more accessible. Recent works [27,28] combine MediaPipe landmarks with CNN-LSTM hybrid architectures, achieving competitive recognition accuracy with significantly reduced computational cost compared to methods operating directly on raw RGB frames. Object detection frameworks like YOLO have also been adapted for hand gesture detection in sign language recognition [29]. Comprehensive surveys [30,31] provide thorough analysis of the relative strengths and weaknesses of these diverse approaches.

2.2. Gloss-Free Sign Language Translation

A critical distinction in SLT is between gloss-based and gloss-free approaches. Gloss-based methods use intermediate gloss representations as a bridge between the visual and textual domains. The seminal work of Camgöz et al. [2] introduced the first neural SLT system using a CNN-RNN encoder-decoder architecture with gloss supervision. Subsequent works [4] employed Transformer architectures for improved sequence-to-sequence translation.

Gloss-free methods bypass the gloss bottleneck by learning direct mappings from video to text. GFSLT-VLP [6] leverages visual-language pretraining to bridge the modality gap without gloss supervision. GASLT [32] introduces a gloss attention mechanism that implicitly discovers sign units without explicit gloss annotations. These methods are increasingly preferred because: (i) gloss annotation is prohibitively expensive for large-scale datasets; (ii) glosses are not standardized across different sign languages; and (iii) gloss representations inherently lose information about non-manual features and prosodic elements that are critical for accurate translation.

2.3. Large Language Models and Vision–Language Models for Sign Language

The integration of LLMs into SLT represents the most significant recent development in the field. As foundation models for natural language understanding have scaled dramatically in both parameters and capabilities [9], researchers have explored increasingly sophisticated approaches to connect visual sign language representations with the powerful text generation capabilities of these models.

SignLLM [10] introduces a Vector-Quantized Visual Sign (VQ-Sign) module that discretizes continuous visual features into a codebook of sign tokens, which are then processed by a frozen LLM. This approach benefits from the discrete nature of the representation, which aligns well with the LLM’s tokenized input format, but may lose fine-grained continuous visual information during quantization.

Sign2GPT [11] represents a complementary approach that employs pseudo-gloss pretraining to teach visual encoders meaningful sign representations before connecting them to a frozen GPT model via lightweight linear adapters. By first learning a pseudo-gloss vocabulary through unsupervised clustering, Sign2GPT creates an intermediate representation that bridges the gap between visual features and language tokens.

SpaMo [12] proposes an efficient approach using separate spatial and motion encoders to capture complementary visual features. The spatial encoder processes individual frames for appearance understanding, while the motion encoder captures inter-frame dynamics. These features are integrated via a Sign Adapter and processed by an LLM with LoRA fine-tuning. SpaMo achieves strong performance on PHOENIX-14T but does not incorporate structural skeletal information.

MMSLT [13] takes a different approach by leveraging MLLMs to generate textual descriptions of sign language components, including hand shapes, body postures, and facial expressions. These descriptions are then fused with video features for translation. This method uniquely exploits the visual understanding capabilities of MLLMs but introduces additional computational overhead from the description generation step.

SCOPE [14] introduces context-aware SLT for dialogue scenarios by incorporating conversational history into the translation process. SignAlignLM [33] focuses on natively integrating sign language support into LLMs through multitasking paradigms.

Contrastive Learning Approaches. SignCLIP [15] and CLIP-SLA [34] explore contrastive learning to create shared embedding spaces between sign language videos and text, drawing on the successful CLIP paradigm. These approaches demonstrate that alignment-based pretraining can significantly improve downstream SLT performance.

More broadly, multimodal vision–language models have evolved rapidly as foundational architectures for tasks requiring joint visual and textual reasoning [35], providing the theoretical and practical grounding for adapting such models to the sign language domain.

Gap Analysis. Despite the significant progress outlined above, no existing LLM-based SLT approach effectively combines both visual (RGB) and skeletal (pose) modalities through deep cross-modal attention mechanisms. SpaMo uses spatial and motion features from the same visual modality, while MMSLT generates textual descriptions rather than performing feature-level fusion. Our proposed SignFuse architecture addresses this gap by introducing bidirectional cross-attention between complementary visual and skeletal feature streams, drawing on the architectural principles established across the broader vision–language model landscape [9,35].

2.4. Multimodal Fusion Strategies

Cross-modal fusion has been extensively studied in general vision-language understanding [36,37,38]. Fusion strategies can be broadly categorized as follows:

Early Fusion concatenates raw or low-level features from different modalities before processing, which is simple but fails to capture modality-specific patterns before fusion.

Late Fusion processes each modality independently through separate encoders and combines predictions at the decision level, which preserves modality-specific representations but cannot model cross-modal interactions.

Cross-Attention Fusion employs attention mechanisms [21] to enable one modality to attend to information in another, allowing dynamic, input-dependent feature combination. This approach has proven particularly effective in models like Flamingo [38] and ALBEF [37].

Our CMFA module adopts a bidirectional cross-attention design that combines the advantages of cross-attention fusion with a learned gating mechanism, specifically designed for the unique requirements of visual-skeletal fusion in sign language where both modalities provide complementary but partially overlapping information.

3. Proposed Methodology: SignFuse

3.1. Problem Formulation

Given a sign language video $\mathbf{V}=\{v_1,v_2,\dots ,v_T\}$ consisting of T RGB frames, where each frame $v_t\in {\mathbb{R}}^{3\times H\times W}$, and corresponding skeleton sequences $\mathbf{S}=\{s_1,s_2,\dots ,s_T\}$, where each skeleton $s_t\in {\mathbb{R}}^{V\times C}$ contains V joint coordinates in C dimensions, our goal is to generate a spoken language sentence $\mathbf{Y}=\{y_1,y_2,\dots ,y_N\}$ with N tokens. This can be formulated as maximizing the conditional probability:

$${\mathbf{Y}}^{*}=arg\underset{\mathbf{Y}}{max}P\left(\mathbf{Y}\right|\mathbf{V},\mathbf{S};\mathsf{\Theta })$$

(1)

where $\mathsf{\Theta }$ denotes all learnable parameters. As illustrated in Figure 1, the SignFuse framework decomposes this problem into five components: (1) a Visual Feature Stream, (2) a Skeletal Feature Stream, (3) a Cross-Modal Fusion Attention (CMFA) module, (4) a Hierarchical Temporal Aggregation (HTA) module, and (5) a Sign-Language Adapter (SLA) connected to a frozen LLM.

3.2. Visual Feature Stream

The visual stream processes raw RGB frames to capture appearance-level features including hand shapes, finger configurations, facial expressions, and spatial layouts. We propose utilizing a pre-trained ResNet-50 [39] as the backbone, chosen for its strong feature extraction capabilities, widespread availability, and computational efficiency.

For each frame $v_t$, we extract a spatial feature vector from the global average pooling layer:

$${\mathbf{F}}_v^t=\mathrm{GAP}\left(\mathrm{ResNet}50\left(v_t\right)\right)\in {\mathbb{R}}^{d_v}$$

(2)

where $d_v=2048$ is the output dimension of ResNet-50’s final convolutional block. The complete visual feature sequence is ${\mathbf{F}}_v=\{{\mathbf{F}}_v^1,{\mathbf{F}}_v^2,\dots ,{\mathbf{F}}_v^T\}\in {\mathbb{R}}^{T\times d_v}$.

To reduce computational cost while preserving temporal information, we apply a 1D temporal convolution with batch normalization and activation:

$${\widehat{\mathbf{F}}}_v=\mathrm{ReLU}\left(\mathrm{BN}\left({\mathrm{Conv}1\mathrm{D}}_{k=3,s=2}\left({\mathbf{F}}_v\right)\right)\right)\in {\mathbb{R}}^{T^{\prime }\times d}$$

(3)

where $T^{\prime }=\lceil T/2\rceil$ is the temporally downsampled length, $d=512$ is the unified feature dimension, and $s=2$ denotes the stride. This projection simultaneously reduces the temporal resolution and aligns the feature dimension with the skeletal stream for downstream fusion.

3.3. Skeletal Feature Stream

The skeletal stream captures structural and kinematic information about body joint configurations and their temporal dynamics. Unlike RGB features that encode appearance, skeletal features explicitly represent the spatial topology of the signer’s body, providing invariance to visual nuisances such as clothing, lighting, and background clutter.

Pose Extraction. We propose using MediaPipe Holistic [26] to extract 2D landmarks from each frame, yielding 21 hand landmarks per hand (42 total), 33 body pose landmarks, and optionally 468 face mesh landmarks. For computational efficiency and focusing on the most translation-relevant signals, we select $V=75$ key joints: both hands (42), upper body pose (23), and key facial landmarks (10). Each joint is represented by its $(x,y)$ normalized coordinates, giving $s_t\in {\mathbb{R}}^{75\times 2}$.

Graph Construction. We construct a spatial-temporal skeleton graph $\mathcal{G}=(\mathcal{V},{\mathcal{E}}_s\cup {\mathcal{E}}_t)$ where $\mathcal{V}$ represents the set of body joints, ${\mathcal{E}}_s$ represents spatial edges following the natural skeleton connectivity (bone connections), and ${\mathcal{E}}_t$ represents temporal edges connecting the same joint across consecutive frames.

ST-GCN Encoder. We employ a multi-layer Spatial-Temporal Graph Convolutional Network [25] to process the skeleton sequences. Each ST-GCN layer performs spatial graph convolution followed by temporal convolution:

$${\mathbf{F}}_s^{(l+1)}=\sigma \left(\sum _{k=1}^K{\tilde{\mathbf{D}}}_k^{-{\displaystyle \frac{1}{2}}}{\tilde{\mathbf{A}}}_k{\tilde{\mathbf{D}}}_k^{-{\displaystyle \frac{1}{2}}}{\mathbf{F}}_s^{\left(l\right)}{\mathbf{W}}_k^{\left(l\right)}\right)$$

(4)

where ${\tilde{\mathbf{A}}}_k={\mathbf{A}}_k+\mathbf{I}$ is the k-th normalized adjacency matrix partition with self-connections, ${\tilde{\mathbf{D}}}_k$ is the corresponding degree matrix, ${\mathbf{W}}_k^{\left(l\right)}\in {\mathbb{R}}^{d_{\mathrm{in}}\times d_{\mathrm{out}}}$ are learnable parameters, and $\sigma$ denotes the ReLU activation function. We use $K=3$ spatial partitions (self, inward, outward neighbors).

We stack $L=4$ ST-GCN blocks with progressively increasing channel dimensions $[2,64,128,256,512]$, with a temporal stride of 2 in the third block to align the temporal resolution with the visual stream. Joint-level features are aggregated via global average pooling over the joint dimension:

$${\widehat{\mathbf{F}}}_s={\mathrm{AvgPool}}_{\mathrm{joints}}\left(\mathrm{STGCN}\left(\mathbf{S}\right)\right)\in {\mathbb{R}}^{T^{\prime }\times d}$$

(5)

3.4. Cross-Modal Fusion Attention (CMFA)

The core theoretical contribution of SignFuse is the CMFA module (Figure 2), designed to perform deep bidirectional cross-modal fusion between visual and skeletal features. Unlike simple concatenation or element-wise addition used in baseline models, CMFA enables each modality to selectively attend to complementary information in the other modality, producing enriched representations that capture cross-modal correlations.

Multi-Head Cross-Attention. For each temporal position, we compute queries, keys, and values from both modalities using separate linear projections:

$$\begin{array}{cc}\hfill {\mathbf{Q}}_v& ={\widehat{\mathbf{F}}}_v{\mathbf{W}}_Q^v,{\mathbf{K}}_v={\widehat{\mathbf{F}}}_v{\mathbf{W}}_K^v,{\mathbf{V}}_v={\widehat{\mathbf{F}}}_v{\mathbf{W}}_V^v\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\mathbf{Q}}_s& ={\widehat{\mathbf{F}}}_s{\mathbf{W}}_Q^s,{\mathbf{K}}_s={\widehat{\mathbf{F}}}_s{\mathbf{W}}_K^s,{\mathbf{V}}_s={\widehat{\mathbf{F}}}_s{\mathbf{W}}_V^s\hfill \end{array}$$

(7)

where ${\mathbf{W}}_Q^{(·)},{\mathbf{W}}_K^{(·)},{\mathbf{W}}_V^{(·)}\in {\mathbb{R}}^{d\times d}$ are learnable projection matrices. The bidirectional cross-attention is computed as:

$$\begin{array}{cc}\hfill {\mathbf{O}}_{v\to s}& =\mathrm{softmax}\left({\displaystyle \frac{{\mathbf{Q}}_v{\mathbf{K}}_s^{\top }}{\sqrt{d_k}}}\right){\mathbf{V}}_s\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\mathbf{O}}_{s\to v}& =\mathrm{softmax}\left({\displaystyle \frac{{\mathbf{Q}}_s{\mathbf{K}}_v^{\top }}{\sqrt{d_k}}}\right){\mathbf{V}}_v\hfill \end{array}$$

(9)

where $d_k=d/h$ is the per-head dimension and $h=8$ is the number of attention heads. ${\mathbf{O}}_{v\to s}$ represents visual features enriched by attending to skeletal information (“what does the skeleton tell me about what I’m seeing?”), while ${\mathbf{O}}_{s\to v}$ represents skeletal features enriched by visual context (“what does the visual appearance tell me about the joint configuration?”).

Gated Aggregation. Rather than simply concatenating or averaging the two cross-attended outputs, we employ a learned gating mechanism that dynamically determines the optimal mixing ratio for each temporal position:

$$\begin{array}{cc}\hfill \alpha & =\sigma ({\mathbf{W}}_g[{\mathbf{O}}_{v\to s};{\mathbf{O}}_{s\to v}]+b_g)\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\mathbf{F}}_{\mathrm{fused}}& =\mathrm{LN}(\alpha ·{\mathbf{O}}_{v\to s}+(1-\alpha )·{\mathbf{O}}_{s\to v})+{\mathbf{F}}_{\mathrm{res}}\hfill \end{array}$$

(11)

where $\sigma$ is the sigmoid function, ${\mathbf{W}}_g\in {\mathbb{R}}^{2d\times d}$ is the gate projection, LN denotes Layer Normalization, and ${\mathbf{F}}_{\mathrm{res}}=({\widehat{\mathbf{F}}}_v+{\widehat{\mathbf{F}}}_s)/2$ is the residual connection ensuring gradient flow and information preservation.

Feed-Forward Network. Following standard Transformer practice, we apply a position-wise feed-forward network after the fusion:

$${\mathbf{F}}_{\mathrm{fused}}^{\prime }=\mathrm{LN}({\mathbf{F}}_{\mathrm{fused}}+\mathrm{FFN}\left({\mathbf{F}}_{\mathrm{fused}}\right))$$

(12)

where $\mathrm{FFN}\left(\mathbf{x}\right)={\mathbf{W}}_2·\mathrm{GELU}({\mathbf{W}}_1\mathbf{x}+b_1)+b_2$ with inner dimension $4d=2048$.

We stack $L=3$ CMFA layers to enable progressively deeper cross-modal interactions. After the first layer, both inputs are derived from the fused representation, allowing the model to iteratively refine the cross-modal alignment.

3.5. Hierarchical Temporal Aggregation (HTA)

Sign language conveys meaning at multiple temporal granularities, which necessitates multi-scale temporal modeling. We identify three distinct temporal scales:

• Frame-level (fine-grained): Individual hand configurations, finger spelling, and instantaneous facial expressions occur at the frame level (25–30ms per frame at 30fps). These are critical for distinguishing between signs with similar global motions but different hand shapes.
• Segment-level (mid-grained): Individual sign units typically span 5–30 frames (150–1000ms). Capturing segment-level dynamics enables recognition of transitional movements, sign boundaries, and phonological components.
• Sequence-level (coarse-grained): Sentence-level structures, discourse patterns, and long-range dependencies extend across hundreds of frames. Understanding these is essential for generating grammatically correct, contextually appropriate translations.

Given the fused features ${\mathbf{F}}_{\mathrm{fused}}\in {\mathbb{R}}^{T^{\prime }\times d}$, we apply depthwise separable temporal convolutions at three scales:

$$\begin{array}{cc}\hfill {\mathbf{H}}_{\mathrm{frame}}& ={\mathrm{DWConv}1\mathrm{D}}_{k=1}\left({\mathbf{F}}_{\mathrm{fused}}\right)\in {\mathbb{R}}^{T^{\prime }\times d}\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathbf{H}}_{\mathrm{seg}}& ={\mathrm{DWConv}1\mathrm{D}}_{k=5,s=2}\left({\mathbf{F}}_{\mathrm{fused}}\right)\in {\mathbb{R}}^{T^{\prime }/2\times d}\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathbf{H}}_{\mathrm{seq}}& ={\mathrm{DWConv}1\mathrm{D}}_{k=11,s=4}\left({\mathbf{F}}_{\mathrm{fused}}\right)\in {\mathbb{R}}^{T^{\prime }/4\times d}\hfill \end{array}$$

(15)

We employ adaptive average pooling to align all scales to a uniform temporal length $T^{\prime \prime }=64$, then concatenate along the feature dimension:

$${\mathbf{H}}_{\mathrm{multi}}=[{\mathrm{Pool}}_{T^{\prime \prime }}\left({\mathbf{H}}_{\mathrm{frame}}\right);{\mathrm{Pool}}_{T^{\prime \prime }}\left({\mathbf{H}}_{\mathrm{seg}}\right);{\mathrm{Pool}}_{T^{\prime \prime }}\left({\mathbf{H}}_{\mathrm{seq}}\right)]$$

(16)

A linear projection combines the multi-scale features, followed by multi-head temporal self-attention for scale integration:

$${\mathbf{H}}_{\mathrm{final}}=\mathrm{LN}\left({\mathbf{H}}_{\mathrm{combined}}+\mathrm{MHSA}\left({\mathbf{H}}_{\mathrm{combined}}\right)\right)\in {\mathbb{R}}^{T^{\prime \prime }\times d}$$

(17)

3.6. Sign-Language Adapter and LLM Integration

The Sign-Language Adapter (SLA) maps the fused multimodal features into the LLM’s input embedding space. It consists of a temporal compression stage followed by a two-layer MLP projection:

$$\begin{array}{cc}\hfill \mathbf{Z}& ={\mathrm{AdaptivePool}}_{N_s}\left({\mathbf{H}}_{\mathrm{final}}\right)\in {\mathbb{R}}^{N_s\times d}\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill {\mathbf{E}}_{\mathrm{sign}}& ={\mathbf{W}}_2^{\prime }·\mathrm{GELU}({\mathbf{W}}_1^{\prime }·\mathbf{Z}+b_1^{\prime })+b_2^{\prime }\in {\mathbb{R}}^{N_s\times d_{\mathrm{LLM}}}\hfill \end{array}$$

(19)

where $N_s=32$ is the number of sign tokens and $d_{\mathrm{LLM}}=4096$ for LLaMA-2 7B. Layer normalization is applied after the final projection to stabilize the input distribution.

LLM Integration. We propose utilizing LLaMA-2 7B [7] as the language model backbone with LoRA [16] for parameter-efficient adaptation. LoRA introduces low-rank matrices $\mathbf{A}\in {\mathbb{R}}^{d\times r}$ and $\mathbf{B}\in {\mathbb{R}}^{r\times d}$ into the attention layers, where $r\ll d$ is the rank (we use $r=16$, $\alpha =32$). The sign embeddings are prepended to a text prompt and fed to the LLM:

$$P\left(\mathbf{Y}\right|\mathbf{V},\mathbf{S})=\prod _{n=1}^{N}P\left(y_n\right|{\mathbf{E}}_{\mathrm{sign}},{\mathbf{e}}_{\mathrm{prompt}},y_{<n};\mathsf{\Theta })$$

(20)

where ${\mathbf{e}}_{\mathrm{prompt}}$ encodes the instruction “Translate the following sign language video into a spoken language sentence:” and $\mathsf{\Theta }$ includes both the frozen LLM parameters and the trainable LoRA parameters.

3.7. Progressive Multi-Stage Training

We propose a three-stage progressive training strategy (Figure 3) to systematically bridge the gap between multimodal sign language features and the linguistic space of the LLM. Each stage addresses a specific modality gap.

Stage 1: Visual-Skeletal Contrastive Pre-training. The first stage trains the dual-stream encoders and CMFA to produce aligned visual-skeletal representations. We employ an InfoNCE contrastive loss [15]:

$$\begin{array}{c}{\mathcal{L}}_{\mathrm{contra}}=-{\displaystyle \frac{1}{2B}}\sum _{i=1}^B[log{\displaystyle \frac{e^{\mathrm{sim}({\mathbf{f}}_v^i,{\mathbf{f}}_s^i)/\tau }}{{\sum }_j{e}^{\mathrm{sim}({\mathbf{f}}_v^i,{\mathbf{f}}_s^j)/\tau }}}\hfill \\ \hfill +log{\displaystyle \frac{e^{\mathrm{sim}({\mathbf{f}}_s^i,{\mathbf{f}}_v^i)/\tau }}{{\sum }_j{e}^{\mathrm{sim}({\mathbf{f}}_s^i,{\mathbf{f}}_v^j)/\tau }}}]\end{array}$$

(21)

where $\mathrm{sim}(·,·)$ is cosine similarity, $\tau =0.07$ is the temperature parameter, B is the batch size, and ${\mathbf{f}}_v^i$, ${\mathbf{f}}_s^i$ are pooled visual and skeletal features. The LLM remains frozen in this stage. Trainable: CNN backbone, ST-GCN, CMFA.

Stage 2: Multimodal-Text Alignment. With frozen feature encoders, this stage trains the CMFA, HTA, and SLA modules to map fused sign features into the LLM’s embedding space:

$${\mathcal{L}}_{\mathrm{align}}={\parallel {\mathbf{E}}_{\mathrm{sign}}-\mathrm{sg}\left({\mathbf{E}}_{\mathrm{text}}\right)\parallel }_2^2+{\mathcal{L}}_{\mathrm{LM}}$$

(22)

where $\mathrm{sg}(·)$ denotes stop-gradient, ${\mathbf{E}}_{\mathrm{text}}$ are the LLM’s text embeddings for the target sentence, and ${\mathcal{L}}_{\mathrm{LM}}$ is the standard autoregressive language modeling loss. Trainable: CMFA, HTA, SLA. Frozen: CNN, ST-GCN, LLM.

Stage 3: End-to-End Fine-tuning. Finally, we unfreeze all trainable components and apply LoRA to the LLM:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{LM}}+{\lambda }_1{\mathcal{L}}_{\mathrm{contra}}+{\lambda }_2{\mathcal{L}}_{\mathrm{align}}$$

(23)

where ${\lambda }_1=0.1$ and ${\lambda }_2=0.05$ are balancing coefficients determined by preliminary analysis. Trainable: All components + LoRA adapters. Frozen: LLM backbone weights.

4. Implementation Specifications

To facilitate reproducibility and community adoption, we provide complete implementation specifications.

4.1. Model Configuration

Table 1 summarizes the proposed model configuration. The total trainable parameter count (excluding the frozen LLM backbone) is approximately 65.9M parameters, making SignFuse comparable in overhead to existing methods like SpaMo and MMSLT.

4.2. Proposed Training Hyperparameters

Table 2 details the proposed training configuration for each stage. All stages use AdamW optimizer with weight decay $0.01$ and cosine annealing with linear warmup.

4.3. Software Availability

We provide a complete PyTorch implementation consisting of:

• model.py: Full architecture implementation including all five proposed components (VisualStream, SkeletalStream, CMFA, HTA, SLA), loss functions (InfoNCE, Alignment, Combined), and a model builder utility. The implementation has been validated for correct tensor shapes through a comprehensive forward-pass test.
• train.py: Complete three-stage training pipeline with stage-specific parameter freezing, optimizer configuration, learning rate scheduling, checkpoint management, and synthetic data fallback for pipeline testing.
• evaluate.py: Evaluation utilities implementing BLEU-1/2/3/4, ROUGE-L, and METEOR metrics from scratch, along with visualization tools for generating comparison charts.

5. Proposed Experimental Protocol

In this section, we formalize the experimental methodology that we invite the community to execute.

5.1. Target Datasets

We propose evaluating SignFuse on three widely-used SLT benchmarks that span different sign languages, vocabulary sizes, and difficulty levels:

PHOENIX-14T [2]: A German Sign Language (DGS) dataset containing 8,257 aligned video-text pairs from weather broadcasts. The dataset is split into 7,096 training, 519 development, and 642 test samples. Videos feature a single signer with a clean background, making it the most controlled benchmark.

CSL-Daily [40]: A large-scale Chinese Sign Language dataset with 20,654 video-sentence pairs covering daily life topics. The dataset features 10 signers and more diverse content than PHOENIX-14T, providing a stronger test of generalization.

How2Sign [41]: A large-scale American Sign Language (ASL) dataset with over 35,000 video clips covering instructional content. Due to its diverse topics, multiple signers, and varying backgrounds, it represents the most challenging benchmark.

5.2. Evaluation Metrics

Following standard evaluation protocols for SLT, we recommend reporting:

• BLEU-n ($n\in \{1,2,3,4\}$) [42]: Measures n-gram precision between hypothesis and reference translations with a brevity penalty. BLEU-4 is the primary metric for comparison.
• ROUGE-L [43]: Measures the longest common subsequence between hypothesis and reference, capturing sentence-level fluency.
• METEOR [44]: Combines precision, recall, and alignment with synonym matching for more comprehensive evaluation.

5.3. Baseline Methods for Comparison

We recommend comparing against the following state-of-the-art methods, whose results are publicly available:

Table 3. Existing SOTA results on PHOENIX-14T (gloss-free, test set) to compare against. B@4 = BLEU-4, R-L = ROUGE-L.

Method	B@4	R-L
GFSLT-VLP [6] (ICCV 2023)	21.44	43.78
Sign2GPT [11] (2024)	22.52	44.83
SignLLM [10] (CVPR 2024)	23.51	46.35
SpaMo [12] (NAACL 2025)	24.32	47.56
MMSLT [13] (ICCV 2025)	25.18	48.34

Table 3. Existing SOTA results on PHOENIX-14T (gloss-free, test set) to compare against. B@4 = BLEU-4, R-L = ROUGE-L.

Method	B@4	R-L
GFSLT-VLP [6] (ICCV 2023)	21.44	43.78
Sign2GPT [11] (2024)	22.52	44.83
SignLLM [10] (CVPR 2024)	23.51	46.35
SpaMo [12] (NAACL 2025)	24.32	47.56
MMSLT [13] (ICCV 2025)	25.18	48.34

5.4. Proposed Ablation Studies

To validate the contribution of each component, we recommend the following ablation experiments on the PHOENIX-14T development set:

• Component ablation: Progressively add components (Visual-only → +Skeletal → +CMFA → +HTA → +Progressive Training) to measure incremental gains.
• Fusion strategy comparison: Compare CMFA against concatenation, element-wise addition, unidirectional cross-attention (V→S and S→V), and bidirectional without gating.
• Temporal scale analysis: Evaluate different combinations of HTA temporal scales to identify the most impactful granularities.
• Training stage analysis: Measure the benefit of each progressive training stage compared to direct end-to-end training.
• LLM backbone sensitivity: Test with different LLM backbones (Flan-T5, Vicuna, LLaMA-2 7B/13B) to assess the framework’s generalizability.

5.5. Expected Theoretical Advantages

Based on our architectural analysis, we hypothesize that SignFuse will demonstrate significant improvements over existing methods for the following reasons:

Complementary modality fusion. RGB features and skeletal features provide complementary information channels. When motion blur degrades RGB features (a well-documented limitation [30]), skeletal landmarks remain stable, and vice versain conditions of partial occlusion where pose estimation may fail, RGB features can still capture hand shapes. The CMFA module’s bidirectional attention enables the model to dynamically leverage whichever modality is more reliable for each temporal position.

Multi-scale temporal understanding. The HTA module’s hierarchical design aligns with the linguistic structure of sign language. Frame-level features should improve fingerspelling accuracy, segment-level features should capture individual sign boundaries, and sequence-level features should improve the translation of compound sentences and temporal modifiersa common failure mode in existing systems [12].

Progressive training stability. By decomposing the training into three stages, each addressing a specific modality gap, we expect improved optimization stability and final performance compared to direct end-to-end training, which must simultaneously learn cross-modal alignment and language generation.

6. Discussion

6.1. Theoretical Analysis of CMFA

The CMFA module can be understood as computing a soft correspondence between visual and skeletal features at each temporal position. The attention weights $\mathrm{softmax}({\mathbf{Q}}_v{\mathbf{K}}_s^{\top }/\sqrt{d_k})$ represent the degree to which each visual feature should attend to each skeletal feature, effectively creating a learned visual-skeletal alignment. This is particularly important for sign language because the same hand shape may appear in different spatial positions (captured by skeleton) with different orientations (captured by appearance), and vice versa.

The gating mechanism $\alpha$ provides an additional degree of flexibility, allowing the model to emphasize visual features when appearance information is more discriminative (e.g., for distinguishing similar hand shapes) and skeletal features when structural information is more informative (e.g., for encoding spatial relationships between both hands). We hypothesize that the learned gate values will exhibit interpretable patterns correlating with sign linguistic categories.

6.2. Computational Cost Analysis

Table 4 provides a theoretical comparison of computational costs. While SignFuse introduces additional parameters from the skeletal stream and fusion modules, the overhead is modest relative to the LLM backbone.

6.3. Limitations and Future Directions

We acknowledge the following limitations of the current work:

• Pending empirical validation. The most significant limitation is the absence of empirical results. While the architecture is fully specified and the codebase is complete, the claims remain theoretical until validated on benchmark datasets with real training.
• Pose estimation reliability. Our reliance on MediaPipe for skeleton extraction means that the skeletal stream’s effectiveness is bounded by MediaPipe’s accuracy, which can degrade under poor lighting, occlusion, or non-frontal camera angles.
• Fixed temporal scales. The HTA module uses fixed kernel sizes $\{1,5,11\}$, which may not be optimal for all signing speeds and styles. Adaptive kernel selection could improve robustness.
• Single sign language. While we propose evaluation on multiple datasets, each represents a different sign language. Cross-lingual transfer between sign languages remains an open challenge.

Future work should explore: (i) adaptive pose estimation with learned error correction; (ii) dynamic temporal scaling based on signing speed; (iii) explicit incorporation of non-manual features (mouthing, eye gaze) as a third modality; and (iv) multi-lingual sign language translation leveraging shared skeletal representations across sign languages.

7. Open Call to the Research Community

The computational hardware required to validate high-parameter multimodal LLMsoften requiring continuous access to A100/H100 GPU clusters over several daysremains a severe bottleneck for many research groups. This resource asymmetry creates a gap between architectural innovation and empirical validation that we aim to bridge through open collaboration.

This paper serves as both a theoretical hypothesis and an open-source collaboration request. We have constructed the complete mathematical framework, detailed architectural specifications, and a fully functional PyTorch codebase for SignFuse. We cordially invite research laboratories and academic partners who possess the requisite computational resources to:

• Execute the proposed training pipeline on PHOENIX-14T, CSL-Daily, and/or How2Sign using the provided codebase and hyperparameters.
• Report empirical results following the evaluation protocol defined in Section 5, including all recommended ablation studies.
• Extend or modify the architecture based on empirical findings, e.g., exploring alternative backbone networks, different fusion strategies, or additional modalities.
• Collaborate on joint publications reporting the empirical validation results.

We request that researchers who validate, falsify, or modify the SignFuse hypothesis cite this foundational architectural blueprint. We believe that open, collaborative research practiceswhere theoretical contributions and empirical validations are distributed across groups with complementary resourceswill accelerate progress in sign language translation and accessibility technology.

8. Conclusion

We have presented the complete theoretical blueprint for SignFuse, a proposed dual-stream cross-modal fusion framework for gloss-free sign language translation with Large Language Models. By formalizing the mathematical integration of a ResNet-based visual stream with an ST-GCN-based skeletal stream via our novel Cross-Modal Fusion Attention (CMFA) module, and proposing a Hierarchical Temporal Aggregation (HTA) mechanism for multi-scale temporal modeling, SignFuse offers a principled solution to the limitations found in existing single-modality SLT systems.

Our progressive three-stage training strategy provides a systematic approach to bridging the gap between multimodal sign language features and the linguistic space of LLMs, while the complete PyTorch implementation ensures that the proposed architecture can be immediately deployed for empirical validation. While the empirical evaluation is left to future community collaboration, the structural foundations, mathematical rigor, and implementation detail established herein offer a compelling and reproducible paradigm for advancing sign language translation technology toward more accurate, inclusive, and practical systems for the Deaf and hard-of-hearing community.