S3-S2ST: Semantically Supervised Streaming Speech-to-Speech Translation
cs.SD
Submitted: 2026-10-27
Updated: 2026-11-12
Code: https://github.com/byteresearchcla/RealSI
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Streaming speech-to-speech translation requires timely decisions about when partial input is sufficient for accurate translation.
Abstract
Streaming speech-to-speech translation requires timely decisions about when partial input is sufficient for accurate translation. We present S3-S2ST, a model that learns to translate and speak from semantically aligned streaming supervision. By converting utterance-level speech pairs into incremental training examples, we teach a learned policy when to commit translations. Its adjustable threshold controls the quality--latency tradeoff on a fixed 640,ms input grid. S3-S2ST delivers competitive English--Chinese translation on FLEURS and leads the evaluated streaming systems in translation quality on the internal 9-pair test set. On RealSI, adapting the threshold to buffered speech further improves quality over a low fixed threshold with little change in first-audio latency. We plan to release the model and code.
Sources
- OpenSTBench: Beyond Semantic Evaluation for Speech Translation
- DIAL-S2ST: Learning Latency Control for Simultaneous Speech-to-Speech Translation
- X-Dubbing: Speech-to-Speech Translation under Temporal and Visual Constraints
- DuraS2ST: Chain-of-Thought and Reinforcement Learning for Duration-Aligned Speech-to-Speech Translation
- Efficient Monotonic Multihead Attention
- Seamless: Multilingual Expressive and Streaming Speech Translation
- Qwen2.5-Omni Technical Report
- Qwen3-Omni Technical Report
- X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation System
Related papers
- Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
- MuLaCover: Flexible Cover Song Generation via Symbolic Tonal Control
- Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems