Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production

The paper proposes Uni-SLTP, a unified framework for sign language translation (SLT) and sign language production (SLP), addressing the challenge of unifying these two tasks that map between continuous sign motions and discrete text tokens in opposite directions. The authors state: SLT maps sign sequences to text, whereas SLP maps text to sign sequences, making it non-trivial to formulate a single unified task for joint modeling. They identify two key challenges: "(1) bridging the modality gap between continuous sign motions and discrete text tokens through a shared sign tokenizer that supports both linguistic abstraction and motion reconstruction; and (2) learning a single conditional autoregressive model that can take either sign or text as input and generate the corresponding target sequence in the opposite modality."

To address these challenges, the paper introduces two key components. First, a Semantic-Reconstruction guided Residual Vector Quantization (SR-RVQ) tokenizer, which is described as a hierarchical tokenizer that decouples semantic alignment from motion-detail reconstruction via a semantic anchor token and residual motion tokens, bridging the Semantic–Reconstruction Gap. The SR-RVQ uses an N-stage residual vector quantizer where the first stage produces an alignment-oriented semantic stream, while later stages encode residual motion details to progressively refine reconstruction. The first-stage tokens are trained with gloss CTC loss and contrastive learning with text, while residual stages are optimized with reconstruction loss. The tokenizer objective is: L tok = λ ctc L ctc + λ con L con + λ rec L rec + λ vq L vq.

Second, the paper proposes a unified autoregressive generation model that formulates both tasks as conditional sequence generation. The framework uses a unified vocabulary combining text tokens and stage-aware pose tokens, and both SLT and SLP are cast as conditional next-token prediction. The training uses a progressive schedule: we first optimize the model on SLP, then on SLT, and finally perform joint training on both tasks. For SLT, only the semantic-stage IDs Q1 are used as source input, while for SLP, all RVQ stages are serialized into a single pose-token sequence.

Experiments are conducted on Phoenix14T and CSL-Daily datasets. For SLT, the paper reports: under the same pose-based setting, Uni-SLTP remains competitive: it achieves performance comparable to the baseline VAP on Phoenix14T and yields an improvement on CSL-Daily. For SLP, the results show: Uni-SLTP outperforms prior methods in both the B-T metric for semantic consistency and the motion-related metrics for motion accuracy. The paper also compares with unified human motion methods, finding that Directly applying these models to sign language tasks results in worse SLT performance and lower SLP performance than our method.

Ablation studies show that the SR-RVQ design is crucial: removing CTC (d) notably hurts SLT, and removing contrastive learning (e) reduces CSLR/SLT. (f) SR-RVQ (Ours) combines residual quantization with both CTC and contrastive supervision, yielding the best overall trade-off. The paper also finds that using all RVQ stages for SLP and only the semantic stage for SLT gives the best results, and that the framework is not sensitive to the choice of pre-trained LM backbone, as different backbones lead to consistent performance with only minor variations.

The paper concludes: "We revisit SLT and SLP as two inverse directions of sign and text mapping, and show that unifying them is feasible only if the sign representation supports both semantic alignment for SLT and motion accuracy for SLP. We propose Uni-SLTP with SR-RVQ, a hierarchical pose tokenizer that converts continuous pose sequences into discrete multi-stage tokens. It separates a semantic anchor token from residual motion detail tokens to bridge the Semantic-Reconstruction Gap. With the resulting shared token space, both tasks reduce to conditional next token prediction and can be handled by a single language model. Experiments on two widely used datasets show stronger SLP performance while keeping pose-based SLT competitive with task-specific baselines."

Improvements to AI Systems:

1. Unified Bidirectional Modality Translation: Build a single autoregressive transformer that handles both sign-to-text (SLT) and text-to-sign (SLP) without task-specific heads. The system uses a shared vocabulary of text tokens and stage-aware pose tokens, enabling it to switch input/output modalities dynamically. This reduces model duplication and allows cross-task knowledge transfer (e.g., learning motion grammar from SLP improves SLT disambiguation).

2. Hierarchical Semantic-Motion Tokenization (SR-RVQ): Implement a multi-stage residual vector quantizer where the first stage produces a semantic anchor token (aligned to text via CTC loss and contrastive learning) and later stages encode fine-grained motion residuals. This decouples high-level meaning from low-level kinematic detail. The improved system can:

• Perform SLT using only the semantic stream (fast, robust to noise).

• Perform SLP using all stages (motion-accurate, realistic).

• Optionally reconstruct missing motion details from semantic tokens alone, enabling partial-input sign understanding.

3. Progressive Joint Training Schedule: Train the unified model first on SLP (text→sign), then on SLT (sign→text), and finally jointly. This curriculum stabilizes learning by first establishing motion priors, then using those priors to ground text semantics. The improved system converges faster and avoids catastrophic forgetting between the two inverse tasks.

4. Stage-Aware Serialization for SLP: Serialize all RVQ stages into a single pose-token sequence for generation, while using only the semantic stage for SLT input. This allows the model to generate high-fidelity motion sequences token-by-token, with early tokens capturing meaning and later tokens refining articulation—enabling controllable generation (e.g., stop early for coarse motion, continue for fine detail).

5. Robustness to Backbone Choice: The framework’s performance is insensitive to the pre-trained language model backbone (e.g., BART, T5, or GPT variants). This means the improved system can be deployed with any off-the-shelf LM, reducing engineering overhead and allowing easy upgrades as better backbones emerge.

6. Ablation-Informed Loss Balancing: Use the ablation findings to set loss weights: CTC loss is critical for SLT, contrastive loss for semantic alignment, and reconstruction loss for motion fidelity. The improved system automatically tunes λ ctc, λ con, λ rec, and λ vq based on task priority (e.g., higher λ ctc for SLT-heavy deployments, higher λ rec for SLP-heavy ones).

What the Improved AI System Can Do:

• Real-time bidirectional sign language translation: A deaf user signs, and the system outputs text; a hearing user types, and the system produces animated sign sequences—all in one model, with no task-switching latency.

• Sign language learning assistant: Given a text sentence, it generates multiple sign motion variants (via residual token sampling) with adjustable semantic vs. motion fidelity.

• Partial sign understanding: From a truncated or noisy sign video, it can still extract the semantic anchor token to infer meaning, even if motion details are lost.

• Cross-lingual sign transfer: Because the semantic tokens align with text embeddings, the system can translate sign from one language to another by generating text in a target language from the semantic stream, then producing sign in that language’s motion style.

• Data-efficient adaptation: New sign languages can be added by fine-tuning only the tokenizer’s residual stages (motion) while keeping the semantic stream and LM frozen, drastically reducing annotation needs.

Recent advances in sign language (SL) research have shown a trend toward unifying multiple sign language understanding (SLU) subtasks, such as isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT), within a single framework, leading to substantial progress. Meanwhile, sign language production (SLP), which generates sign sequences from text, has also attracted growing attention. This naturally raises an important question: can sign language understanding and production be unified within a single framework? Compared with unifying SLU subtasks, this problem is substantially more challenging. Existing SLU tasks largely share the same direction of mapping, namely from sign inputs to linguistic outputs, whereas SLT and SLP lie in opposite directions of sign-text mapping. A unified framework must therefore address two key challenges: (1) bridging the modality gap between continuous sign motions and discrete text tokens through a shared sign tokenizer that supports both linguistic abstraction and motion reconstruction; and (2) learning a single conditional autoregressive model that can take either sign or text as input and generate the corresponding target sequence in the opposite modality. To this end, we propose Uni-SLTP, a unified framework for SLT and SLP with two key components: (1) a shared sign tokenizer that converts sign sequences into discrete tokens and latent representations, capturing both semantic and reconstructive information; and (2) a unified autoregressive generation model that formulates both tasks as conditional sequence generation. Experiments on widely used public datasets show that Uni-SLTP achieves superior motion accuracy for SLP while maintaining competitive SLT performance.