Tracing a Sparse Emotion-Control Circuit in LLM-Based Text-to-Speech
cs.SD, cs.AI
Submitted: 2026-10-04
Updated: 2026-10-04
Comments: Accepted to EMNLP 2026 (Main Conference). 15 pages, 4 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: LLM-based text-to-speech (TTS) models can generate emotionally expressive speech, but how reference emotion is routed through the model and realized in decoded speech remains unclear.
Abstract
LLM-based text-to-speech (TTS) models can generate emotionally expressive speech, but how reference emotion is routed through the model and realized in decoded speech remains unclear. We introduce two emotion-sensitive metrics for matched neutral and emotional syntheses---a codec trajectory score and a late residual direction score---and use them to score activation-patching interventions. Under controlled matched-reference conditions, this analysis identifies a sparse source-to-readout component-level circuit: 23--27 attention heads and MLPs per emotion, roughly 5% of the components considered, recover or suppress 74--88% of the late emotion-readout shift on held-out cases. The circuit combines a shared component backbone with emotion-specific components; cross-emotion activation swaps reduce the target readout in 47 of 48 cases. In decoded speech, the same intervention produces consistent changes in pitch, energy, and spectral brightness over 24 matched pairs per emotion. A readout-matched residual-direction baseline produces only 17--27% of the intervention's pitch effect, showing that internal readout movement alone does not explain the decoded acoustic changes. These results trace a compact causal route from reference-derived prefix information to emotion-relevant properties of generated speech.
Sources
- IndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System
- CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
- Localizing Model Behavior with Path Patching
- How to use and interpret activation patching
- AtP*: An efficient and scalable method for localizing LLM behaviour to components
- Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoders
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
- Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
- EmoSteer-TTS: Fine-Grained and Training-Free Emotion-Controllable Text-to-Speech via Activation Steering
- PROEMO: Prompt-Driven Text-to-Speech Synthesis Based on Emotion and Intensity Control
Related papers
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
- AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
- SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving
- WASIL: In-the-Wild Arabic Spoken Interactions with LLMs