S3-S2ST: Semantically Supervised Streaming Speech-to-Speech Translation

arXiv:2610.36777 · cs.SD · Submitted 2026-10-27 · Read on arXiv

cs.SD

Submitted: 2026-10-27

Updated: 2026-11-12

Code: https://github.com/byteresearchcla/RealSI

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

The gist: Streaming speech-to-speech translation requires timely decisions about when partial input is sufficient for accurate translation.

Abstract

Streaming speech-to-speech translation requires timely decisions about when partial input is sufficient for accurate translation. We present S3-S2ST, a model that learns to translate and speak from semantically aligned streaming supervision. By converting utterance-level speech pairs into incremental training examples, we teach a learned policy when to commit translations. Its adjustable threshold controls the quality--latency tradeoff on a fixed 640,ms input grid. S3-S2ST delivers competitive English--Chinese translation on FLEURS and leads the evaluated streaming systems in translation quality on the internal 9-pair test set. On RealSI, adapting the threshold to buffered speech further improves quality over a low fixed threshold with little change in first-audio latency. We plan to release the model and code.

Sources

Related papers