SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMs

SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMs

Abstract

Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks. However, fine-tuning LLMs for Gloss-Free Sign Language Translation (GFSLT) remains a challenge. In this paper, we investigate how to effectively adapt LLMs to the GFSLT task. We show that there are two key issues that need to be solved: (1) the inherent distributional gap between visual feature inputs and text feature inputs makes it difficult for LLMs to interpret visual inputs; and (2) existing approaches typically concatenate visual and textual features in an autoregressive framework, which leads to the model overemphasizing textual inputs and deprioritizing visual cues, as LLMs are pretrained predominantly on text-centric data. To address the first challenge, we propose a simple yet effective method named Filtered Pseudo-Gloss CTC Pretraining, which leverages filtered pseudo-gloss sequences generated from text sequences to supervise the training of the visual backbone. To tackle the second issue, we introduce a Visual-Prioritized Distillation training strategy. Specifically, we define a visual-only prediction path in which text inputs are masked, and the model is required to generate the target sequence relying solely on visual inputs. To guide this path, the outputs from the standard visual-textual prediction are then distilled into the visual-only prediction path, encouraging the model to prioritize visual features. Comprehensive experiments and qualitative analyses demonstrate the effectiveness of the proposed model. The proposed SignLlama achieves very competitive performance on multiple datasets for GFSLT tasks, without using any extra modalities or external sign language datasets for pretraining.

Introduction

Current sign language (SL) understanding tasks mainly include Continuous SL Recognition (CSLR) and SL Translation (SLT). CSLR aims to recognize a sequence of signs and convert it into the corresponding gloss sequence, while SLT focuses on translating sign sequences into spoken language. As SLT produces more natural and fluent language outputs, it has attracted increasing attention in recent research. The de facto architecture for current SLT tasks typically employs a 2D or 3D CNN-based backbone to extract visual features, followed by temporal modeling modules (e.g., 1D CNNs or LSTMs) to capture dynamic changes across sign frames. Finally, a translation model (e.g., mBART, GPT-2) is used to generate the corresponding spoken language sentence.

SLT research has branched into two distinct paths: gloss-based SLT (GBSLT) and gloss-free SLT (GFSLT). Previous state-of-the-art (SOTA) GBSLT models have emphasized that pretraining the visual encoder with gloss labels is critical for achieving strong SLT performance. Due to the reliance on gloss labels, SLT has increasingly shifted its focus towards GFSLT, which aims to boost SLT performance without gloss supervision.

Meanwhile, Large Language Models (LLMs) have revolutionized the field of natural language processing, setting new benchmarks across a broad spectrum of tasks. Benefiting from the vast pretraining datasets, LLMs demonstrate extraordinary generalization capabilities, and as a result, fine-tuning LLMs for downstream tasks has emerged as a prevailing trend. However, adapting LLMs for the GFSLT task remains understudied, primarily due to the following two key challenges:

1. There exists a fundamental distributional gap between textual features and SL video features. Textual inputs, which are the primary input modality during LLM pretraining, are typically tokenized into discrete, symbolic units, forming clearly segmented sequences. In contrast, SL video features are inherently continuous, with subtle variations between consecutive frames. This discrepancy in representation poses a significant challenge in effectively tokenizing SL videos and providing suitable inputs for LLMs, which is crucial for fine-tuning LLMs for GFSLT.

2. Existing Vision-Language Models (e.g., MiniGPT-4) and LLM-based SLT models typically adopt a straightforward strategy by concatenating visual and textual features directly in an autoregressive manner. However, this approach often leads to the deprioritization of visual features and biases the model toward fitting the distribution of text tokens rather than visual ones, as LLMs are mainly pretrained on text-centric data, and visual inputs constitute an out-of-distribution modality.

To tackle the first challenge, a common approach in vision-language models (VLMs) is to align visual and textual features by employing a pre-trained image encoder and leveraging large-scale image-text datasets with objectives such as Image-Text Contrastive (ITC) loss and Image-Text Matching (ITM) loss. In addition, some works have explored using attention mechanisms, vector quantization, and pseudo glosses to tokenize SL videos. As for the second challenge, the effectiveness of concatenating visual and textual features in an autoregressive way remains underexplored. CAL addresses this by focusing on reweighting text tokens based on their visual relevance, thereby enhancing the alignment between modalities. However, it overlooks the enhancement of visual representations themselves. In contrast, our approach aims to prioritize the contribution of visual features during prediction, ensuring that the model more effectively leverages visual information.

In this paper, we focus on the Gloss-Free SLT task by adopting open-source LLMs (e.g., the Llama family). First, inspired by recent works and GBSLT approaches where CTC loss is commonly used to pretrain the visual encoder, we propose a simple yet effective method to bridge the distributional gap between video and textual features, namely filtered pseudo-gloss CTC Pretraining. Specifically, we generate filtered pseudo-gloss sequences from the target text by applying lemmatization, removing prepositions and conjunctions, and randomly dropping tokens. The resulting pseudo-gloss sequences are then used with CTC loss to pretrain the visual encoder. To further prioritize the model's reliance on visual features, we propose Vision-Prioritized Distillation (VPD). This training strategy distills knowledge from a 'Visual-Textual Prediction' path into a 'Visual-Only Prediction' path, enabling the LLMs to generate text with greater reliance on visual features. By prioritizing visual features, VPD can boost the ability of text generation from visual inputs, thereby improving the performance of GFSLT models.

Contributions

• We introduce a simple yet effective pretraining method, Filtered Pseudo-gloss CTC Pretraining (FPG-CTC), which generates filtered pseudo-gloss sequences to supervise the visual encoder. We show that FPG-CTC can effectively discretize and align SL features, leading to substantial performance improvements on the GFSLT task.

• We propose a Vision-Prioritized Distillation (VPD) training method, which distills the 'Visual-Textual Prediction' into the 'Visual-Only Prediction' to enhance GFSLT performance. The proposed VPD can boost text generation and alleviate the exposure bias by prioritizing visual features.

• We contribute an array of pre-trained SignLlama models to the community, ranging from 1 billion to 13 billion parameters. Extensive experiments on public SL datasets demonstrate the effectiveness of our models, which achieve very competitive performance on the GFSLT task.

Method

Overall Framework. For an SL video with θ frames f = f i i=1 θ, the goal of GFSLT is to generate t = t i i=1 ς with ς words based on input f. The model consists of a visual encoder and a Llama model. The visual encoder VE processes frames to get visual features v=VE(f) with m vectors, where v = v i ∈ R d I i=1 m (m ≤ θ) and d I is the input dimension of Llama. The tokenizer converts the target text into n token IDs u i i=1 n, and the text embedding module maps them to text embeddings e = e i ∈ R d I i=1 n. During training stage, the model predicts token û i based on v and previous ground truth tokens: p(û iv, u j j=0 i-1). During inference, the model predicts token t̂ i based on v and previous predicted tokens: p(û iv, û j j=0 i-1).

Filtered Pseudo-gloss CTC Pretraining

For GBSLT, the visual encoder is typically trained with the CSLR task, in which CTC loss is used to optimize the visual encoder to learn semantic segmentation and alignment with gloss sequences g. A translation model is then integrated and fine-tuned for SLT, taking the output of the encoder as its input. Let R denote the recognition network (with the visual encoder as a subcomponent) and T denote the translation network, while Θ R and Θ T denote the parameters of the recognition and translation networks, respectively. L CTC and L CE denote the CTC loss and the cross-entropy loss. The hyperparameters α and β are used to balance the two objectives.

For gloss-free SLT, the absence of gloss annotations makes it challenging to pretrain the visual encoder. Unlike previous pseudo gloss based GFSLT models that adopt contrastive learning for visual backbone pretraining, we aim to develop a simple yet effective method that enables the visual encoder to effectively tokenize SL videos and align visual features with textual representations. We draw inspiration from previous work and propose the Filtered Pseudo-gloss CTC Pretraining (FPG-CTC) method. Specifically, we generate a pseudo-gloss sequence g p from the target text by applying lemmatization and filtering out prepositions and conjunctions. Considering that certain words in the text may not be explicitly expressed in the corresponding SL video, we further apply random dropping to remove a fixed proportion of words, resulting in the final pseudo-gloss sequence g p.

Visual-Prioritized Distillation Training for LLM Fine-Tuning

After obtaining visual features, a widely adopted approach is to concatenate visual and textual features directly, followed by next-token prediction, which we refer to as 'Visual-Textual Prediction', formalized as: p(t̂ iv, t j j=0 i-1). However, this training paradigm suffers from a critical limitation: visual deprioritization. LLMs tend to prioritize fitting the distribution of text tokens over that of the newly introduced visual tokens. This is because LLMs are pretrained on text-centric data, and visual inputs constitute an out-of-distribution modality, leading to the deprioritization of visual information during generation. During inference, the model no longer has access to ground-truth tokens and instead conditions on its own predictions: p(t̂ iv, t̂ j j=0 i-1). This mismatch between training and testing leads to cumulative errors. When the model is overly reliant on generated text, errors in early predictions can quickly accumulate, leading to degraded performance.

To encourage the LLMs to prioritize visual features, we propose a novel training paradigm termed Visual-Prioritized Distillation. Specifically, we perform two separate forward passes through the LLM: one for Visual-Textual Prediction, where visual and textual features are concatenated as input; and another for Visual-Only Prediction, where the textual inputs are masked out using a 'Visual-Only Mask', and the model is required to generate the full target sequence relying solely on visual inputs. To ensure that the 'Visual-Only Prediction' path can converge and perform well, we distill the logits from the 'Visual-Textual Prediction' path into the 'Visual-Only Prediction' one. This distillation guides the learning process of the visual-only prediction path, encouraging it to align with the output distribution of the visual-textual prediction path while learning to rely exclusively on visual features.

The training loss can be formalized as:

L VPD = L Tvt + L Tv + γ L KD

where γ is a hyperparameter and n denotes the number of tokens. L Tvt, L Tv are cross-entropy losses of 'Visual-Textual Prediction' and 'Visual-Only Prediction', and L KD(T(v,t),T(v)) denotes the distillation loss.

Training and Inference for GFSLT. The final training process of our model begins with pretraining the visual backbone using pseudo-gloss supervision. The entire model is then fine-tuned using the VPD objective together with the FPG-CTC loss. During inference, the visual encoder takes the SL video as input and extracts SL video features, which are then fed into the Llama model. The Llama model follows the standard autoregressive process to generate the target text sequence. Neither pseudo-gloss supervision nor the VPD training procedure is required during inference.

Theoretical Insights into VPD

The difference between VPD and Knowledge Distillation. Although VPD shares a superficial similarity with knowledge distillation (KD), it is conceptually distinct in both purpose and design. The key novelty lies in how VPD prioritizes visual features between visual-only and visual-textual paths, mitigating the tendency of the AR model to over-rely on textual cues and underutilize visual information, which is a critical bottleneck that has not been addressed in prior work. Thus, VPD is not merely an adaptation of KD, but a novel mechanism that explicitly prioritizes visual features in SLT.

The theoretical intuition of VPD. Let v denote the visual input encoded by parameters θ v, t the optional textual input, T(v,t) the teacher (visual-textual path) with logits z vt and softmax probabilities p vt, and S(v) the student (visual-only path) with logits z v and softmax probabilities p v. The simplified VPD loss is:

L = L vt + L v + γ L KD

where L vt = -Σ y i log p vt,i, L v = -Σ y i log p v,i, L KD = KL(p vt ∥ p v) = Σ p vt,i log(p vt,i / p v,i).

The gradient of L with respect to θ v is:

∂L/∂θ v = ∂L vt/∂θ v + ∂L v/∂θ v + γ Σ (p v,i - p vt,i) ∂z v,i/∂θ v

Intuitively, when the teacher assigns a higher probability to the correct class, the KL term provides an additional corrective gradient that pushes p v toward p vt, supplementing the supervision provided by the visual-only CE loss. This provides an additional training signal to the visual encoder, encouraging the visual-only path to produce predictions closer to those of the visual-textual path.

Experiments

Datasets. We evaluate our model on several publicly available SLT datasets, including the widely used Phoenix14T and CSL-Daily, as well as two large-scale ASL datasets: How2Sign and OpenASL.

(1) Phoenix14T is a German Sign Language dataset annotated with both glosses and translations. It consists of 7096 training, 519 validation, and 642 test samples from 9 signers, with a vocabulary of 1066 glosses and 2877 German words.

(2) CSL-Daily is a Chinese SL dataset containing 18401, 1077, and 1176 videos for training, validation, and testing from 10 signers. It provides 2000 glosses and 2343 words for translation.

(3) How2Sign is a large-scale American Sign Language (ASL) dataset comprising over 80 hours of multi-view video and multimodal data. We use only the frontal-view RGB videos, including 31128, 1741, and 2322 samples for training, validation, and testing, respectively.

(4) OpenASL is another large ASL dataset with over 280 hours of video from more than 200 signers. It includes 96476 training, 997 validation, and 999 test samples.

Data Preprocessing. Following prior work, we adopt the same preprocessing pipeline to ensure fair comparisons. During training, we apply standard data augmentations, including resizing frames to 256×256 pixels, random cropping to 224×224 pixels, random horizontal flipping with a probability of 0.5, and random temporal scaling within ±20%. During inference, frames are resized to 256×256 and center-cropped to 224×224.

Implementation Details. Our architecture consists of following key components: (1) Visual Encoder: In our baseline setting, we choose PoolFormer with 1D CNN layers used in previous work. (2) Llama: We adopt the Llama-3.2-1B model in our baseline setting. We choose the base pretrained model rather than instruction-tuned variants, as the effects of prompt engineering are beyond the scope of this work. For LoRA used in Llama, we apply LoRA with a rank of 4 and a scaling factor ('LoRA alph…

Improvements to AI Systems Based on This Paper:

1. Visual-Prioritized Distillation (VPD) for Multimodal LLMs: Implement a dual-path training strategy where one path uses both visual and textual inputs (teacher) and another uses only visual inputs (student). Distill the teacher's logits into the student to force the model to rely more heavily on visual features rather than defaulting to text priors. This reduces over-reliance on text and mitigates exposure bias during autoregressive generation.

2. Filtered Pseudo-Gloss CTC Pretraining (FPG-CTC) for Visual Encoders: Generate pseudo-gloss sequences from target text using lemmatization, stopword removal, and random token dropping. Use these sequences with CTC loss to pretrain the visual backbone, enabling it to produce more discrete, token-like visual representations that align better with LLM text embeddings. This bridges the distributional gap between continuous video features and discrete text tokens.

3. Improved Visual Feature Tokenization for Video-to-Text Tasks: Apply FPG-CTC to any video understanding task (e.g., video captioning, video QA) to make visual features more semantically segmented and aligned with language, improving downstream LLM performance without requiring manual gloss or annotation.

4. Reduced Exposure Bias in Autoregressive Generation: Use VPD to train models to generate from visual inputs alone, reducing cumulative error during inference when ground-truth tokens are unavailable. This improves robustness in long-sequence generation tasks involving multimodal inputs.

5. Cost-Efficient Multimodal Training: VPD adds only 10% training overhead (e.g., 20 to 22 minutes per epoch) while providing significant performance gains, making it practical for resource-constrained settings. This can be adopted in other vision-language models to improve visual grounding without heavy computational costs.

6. Scalable GFSLT Across Languages and Model Sizes: The method works across multiple datasets (Phoenix14T, CSL-Daily, How2Sign, OpenASL) and LLM sizes (1B to 13B), showing it can be generalized to different sign languages and resource levels. This enables deployment of accessible sign language translation systems in low-resource languages.

7. Enhanced Visual Attention in LLMs: VPD demonstrably shifts attention weights toward visual tokens in the final layers, as shown in qualitative analysis. This can be applied to any multimodal LLM to improve visual grounding, interpretability, and trustworthiness in tasks like image/video captioning, visual QA, and robotics instruction following.

What the Improved AI System Can Do:

• Translate sign language videos into fluent spoken language without gloss annotations, across multiple languages (German, Chinese, English) and datasets.

• Generate more accurate and contextually correct translations by prioritizing visual cues over textual priors, reducing hallucination and improving fidelity to the input video.

• Operate efficiently on limited hardware (e.g., single RTX 3090) with minimal training overhead, making it feasible for real-world deployment in assistive technologies.

• Adapt to new sign languages or video-to-text tasks with minimal labeled data, using pseudo-gloss pretraining and distillation to bootstrap visual understanding.

• Provide more reliable outputs in long-form video understanding by mitigating error accumulation during autoregressive decoding.

Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks. However, fine-tuning LLMs for Gloss-Free Sign Language Translation (GFSLT) remains a challenge. In this paper, we investigate how to effectively adapt LLMs to the GFSLT task. We show that there are two key issues that need to be solved: (1) the inherent distributional gap between visual feature inputs and text feature inputs makes it difficult for LLMs to interpret visual inputs; and (2) existing approaches typically concatenate visual and textual features in an autoregressive framework, which leads to the model overemphasizing textual inputs and deprioritizing visual cues, as LLMs are pretrained predominantly on text-centric data. To address the first challenge, we propose a simple yet effective method named Filtered Pseudo-Gloss CTC Pretraining, which leverages filtered pseudo-gloss sequences generated from text sequences to supervise the training of the visual backbone. To tackle the second issue, we introduce a Visual-Prioritized Distillation training strategy. Specifically, we define a visual-only prediction path in which text inputs are masked, and the model is required to generate the target sequence relying solely on visual inputs. To guide this path, the outputs from the standard visual-textual prediction are then distilled into the visual-only prediction path, encouraging the model to prioritize visual features. Comprehensive experiments and qualitative analyses demonstrate the effectiveness of the proposed model. The proposed SignLlama achieves very competitive performance on multiple datasets for GFSLT tasks, without using any extra modalities or external sign language datasets for pretraining.