Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing
Iftach Shoham, Tali Dror, Oren Gal, Haim Permuter, Gilad Katz, Eliya Nachmani
cs.SD, cs.CL, cs.LG
Submitted: 2026-08-05
Updated: 2026-08-10
Code: https://github.com/iftachShoham/SIEDD
License: http://creativecommons.org/licenses/by/4.0/
The gist: Speech recordings often contain missing, corrupted, or incorrect regions that must be reconstructed or modified without re-synthesizing the entire utterance.
Terminology
Abstract
Speech recordings often contain missing, corrupted, or incorrect regions that must be reconstructed or modified without re-synthesizing the entire utterance. Speech inpainting restores missing segments, whereas speech editing replaces spoken content according to an edited transcript. Both tasks require the generated speech to express the intended words while remaining consistent with the surrounding speaker identity, prosody, timing, and recording conditions. Discrete diffusion is particularly well suited to these tasks because it can iteratively refine masked tokens while jointly conditioning on both left and right acoustic context. We introduce SIEDD, a discrete diffusion framework for text-guided speech inpainting and editing over hierarchical codec tokens. Its core architecture, HiCoDD, follows the RVQ generation order by representing previously generated codebooks as clean, committed acoustic context and applying diffusion only to the current refinement codebook. This separation enables leakage-free joint training while matching sequential coarse-to-fine inference. The model further combines phoneme-level conditioning, span-localized classifier-free guidance, and duration prediction to support both fixed-duration inpainting and variable-duration text edits. On the RealEdit benchmark, SIEDD achieves the best overall speech-editing performance among the evaluated methods. It also outperforms the evaluated autoregressive baselines across all speech-inpainting settings, on both single and multiple gaps. These results demonstrate that explicitly modeling the codec hierarchy substantially improves context-preserving speech reconstruction and editing. See our full code at https://github.com/iftachShoham/SIEDD.
Sources
- SpeechPainter: Text-conditioned Speech Inpainting
- GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio
- High Fidelity Neural Audio Compression
- Classifier-Free Diffusion Guidance
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
- FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency
- Speech inpainting: Context-based speech synthesis guided by video
- Transient Noise Removal via Diffusion-based Speech Inpainting
- XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech
- Large Language Diffusion Models
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
- UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
- Score-Based Generative Modeling through Stochastic Differential Equations
- Score-based Continuous-time Discrete Diffusion Models
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
Related papers
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
- Harmonic Representations in Symbolic Music Transformers are Localized and Controllable
- AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
- SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving