Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
summary
The gist
Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints…
In short
TextPro-SLM addresses a modality gap in speech LLMs by making spoken input resemble text from a prosody-aware model. It uses a Speech encoder to generate synchronized text tokens and prosody embeddings, which are then used to train an LLM backbone. This approach achieves the lowest modality gap among leading SLMs with minimal audio training data, proving that aligning speech input is effective.
Key concepts
- Modality Gap
- This refers to the difference in performance between models trained on text and models trained on speech. In TextPro-SLM, the goal is to minimize this gap by ensuring the model can effectively understand both what is said (linguistic content) and how it is said (paralinguistic cues).
- WhisperPro
- This is a unified speech encoder that processes audio input. It generates two crucial outputs simultaneously: text tokens representing the spoken words and prosody embeddings that capture non-verbal aspects like emotion, speaking style, and speaker timbre.
- Coupled Objectives Training
- The LLM backbone is trained using two simultaneous goals: one to maintain the semantic knowledge of a standard text LLM (knowledge distillation) and another to learn how to interpret the prosodic information extracted by WhisperPro. This dual training ensures the model learns both language and speaking style effectively.
Terminology used across episodes
This episode discusses
- Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM · Paper Radio
- LLaMA-Omni: Seamless Speech Interaction with Large Language Models
- Moshi: a speech-text foundation model for real-time dialogue
- TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving · Paper Radio
- GPT-4 Technical Report
- OpenAI GPT-5 System Card
- The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes
- Gemini: A Family of Highly Capable Multimodal Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
- Closing the Gap Between Text and Speech Understanding in LLMs
- GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot
- Spirit LM: Interleaved Spoken and Written Language Model
- Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
- MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
- Qwen3-Omni Technical Report
- Qwen2.5-Omni Technical Report
- Kimi-Audio Technical Report
- Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap · Paper Radio
- Cross-Modal Knowledge Distillation for Speech Large Language Models
- X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs
The paper
Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM · Read on arXiv
Wenqian Cui, Xiao-Hui Li, Daxin Tan, Qiyong Zheng, Irwin King
Chinese University of Hong Kong · Huawei Technologies
Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient. Code is available at https://github.com/dreamtheater123/TextPro-SLM
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Minimizing Modality Gap from the Input Side".
Jane: Here's the gist of the paper. Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap <ref:2605.05927#pg2> The key remaining bottleneck lies on the input side <ref:2605.05927#pg4> TextPro-SLM proposes an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM <ref:2605.05927#pg5> Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales <ref:2605.05927#pg5> This gain is achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient <ref:2605.05927#pg5>.
Tom: Pruning redundant reasoning tokens The paper argues that the remaining bottleneck lies on the input side because under modern thinker-talker designs, the output side already resembles that of a TLM <ref:2605.05927#pg2> The fundamental difficulty is that, unlike a TLM, an SLM must infer both linguistic content and paralinguistic cues directly from speech representations <ref:2605.05927#pg2>.
Jane: The authors propose to minimize the modality gap from the input side by making a Speech LLM behave as closely as possible to a prosody-aware text LLM <ref:2605.05927#pg5> This involves explicitly separating and preserving two complementary aspects of spoken input: what is said and how it is said <ref:2605.05927#pg5>.
Tom: How the training signal is built TextPro-SLM consists of two key components: a Speech encoder, WhisperPro, and an LLM backbone trained with two coupled objectives <ref:2605.05927#pg5> 1) Speech encoder: WhisperPro is a unified speech encoder that produces text tokens that capture utterance content and Prosody embeddings that encode paralinguistic information such as speaking style, emotion, and speaker timbre <ref:2605.05927#pg5>.
Title and authors: Tom: So we're looking at this paper called "Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM". It tackles that problem where speech models, or SLMs, are built from text models, but they still have a big gap between understanding what’s said and understanding how it’s said.
Jane: Right. It suggests the real bottleneck isn't just making the speech output sound more like text anymore; the real issue is how we feed that speech into the model in the first place. They propose TextPro-SLM to fix this by making spoken input look more like what a text model expects, but keeping track of those extra cues.
Lu: Essentially, they argue that since modern SLMs already do a pretty good job of sounding like text when they talk, the input side needs to provide representations that are both compatible with the text model and preserve the way speech is actually delivered. They suggest separating what’s said from how it’s said.
Meng: So, if I'm deploying this, what does that mean practically? Does it mean less data is needed for these models to start getting good at understanding emotion or tone? Because right now I'm worried about the cost of training on massive audio datasets.
Lalam: The paper says that by making the speech encoder produce both text tokens and prosody embeddings, they can achieve the lowest modality gap among leading SLMs at both three billion and seven billion parameter scales <ref:2605.05927#pg5>. It claims this is possible with only about one thousand hours of LLM training audio, which is a lot less than what was previously thought necessary for this level of performance <ref:2605.05927#pg5>.
Tom: one thousand hours for that kind of gap reduction sounds pretty efficient, especially when you look at the results they show on reasoning tasks. The paper claims TextPro-SLM achieves a modality gap of just one point eight percent on high school math, compared to ten point five percent for Qwen2 point 5-Omni-7B and seventeen point five percent for Kimi-Audio-7B <ref:2605.05927#pg5>.
Jane: That’s a big reduction in the gap, Tom. It means that when you use this setup, the model isn't struggling as much to bridge that gap between text and speech representations when it has to do complex reasoning tasks. They also show strong performance on paralinguistic understanding benchmarks <ref:2605.05927#pg5>.
Lu: The methodology involves a Speech encoder called WhisperPro, which generates two things simultaneously: text tokens for the content, and prosody embeddings that capture things like emotion or speaking style <ref:two thousand six hundred five point zero five nine two seven#pg3. Then they train the LLM backbone to keep the original model's smart capabilities while learning to understand those prosodic cues <ref:2605.05927#pg5>.
Title and authors: Meng: I see the two components—the encoder and the backbone—but how do they actually learn together? Is it just one loss function, or are there multiple objectives driving that training process? Because in production, coupling things up is always tricky when you're trying to keep inference fast.
Tom: They use a multi-task objective for the LLM backbone, combining an auto-speech recognition loss and a mel reconstruction loss <ref:2605.05927#pg5>. The final objective they use is L equals ASR loss plus lambda times the mel reconstruction loss, where lambda is set to one point zero <ref:2605.05927#pg5>. That keeps things balanced between making sense of the words and making sense of the acoustic features.
Jane: So, for someone who doesn't read the math, it means they are training the model to do two things at once: accurately transcribe what is being said and reconstruct how that sound was made <ref:2605.05927#pg5>. That’s a clever way to force the model to learn those paralinguistic details without needing separate, huge datasets for them.
Lalam: They specifically highlight that TextPro-SLM achieves the strongest paralinguistic understanding performance among all baselines <ref:2605.05927#pg5>. For example, the seven billion parameter version performs eighty-eight point six on the Gender task and sixty-four point nine on the Age task <ref:2605.05927#pg5>.
Tom: Eighty-eight point six and sixty-four point nine. That’s a solid number for paralinguistic understanding, especially given that they achieved this with only one thousand hours of training audio <ref:two thousand six hundred five point zero five nine two seven#pg5. It shows that even this relatively new approach can capture a lot about *how* something is spoken just by aligning the input representation correctly.
Lu: And it does extend to reasoning tasks, which is where I think this is really interesting for the broader field. They say TextPro-SLM performs substantially better on reasoning-heavy tasks compared to a model called SALAD <ref:2605.05927#pg5>.
Jane: Substantially better means the gap between what it can reason about and what it actually understands is smaller, which is exactly what we want in a practical AI system. They mention TextPro-SLM achieves a modality gap of just one point eight percent on high school math, compared to ten point five percent for Qwen2 point 5-Omni-7B <ref:2605.05927#pg5>.
Meng: If we think about deployment, that low gap suggests that the model is less likely to fail when the input isn't perfectly clean or text-like, which is a huge win for real-world applications where you’re not feeding it pristine text <ref:two thousand six hundred five point zero five nine two seven#pg5. But I still have to wonder about the overhead of running that dual encoder setup in production.
Title and authors: Tom: That overhead is something we need to look at, Meng, but for now, the focus here is on the efficiency gain from using less data while achieving better results across both content understanding and those tricky paralinguistic cues <ref:two thousand six hundred five point zero five nine two seven#pg5.
Lu: The authors also point out that they are focusing on minimizing the modality gap specifically from the input side, arguing that if the output side is already improving, we need to fix how speech gets into the model to support those improvements <ref:two thousand six hundred five point zero five nine two seven#pg2. They explicitly state they are separating and preserving what is said and how it is said <ref:two thousand six hundred five point zero five nine two seven#pg5.
Jane: So, to summarize this whole idea, the core improvement isn't just making the speech sound like text; it’s about giving the model a dual representation—the words plus the prosody—so it can infer both linguistic content and those crucial speaking cues directly from audio <ref:two thousand six hundred five point zero five nine two seven#pg5.
Tom: So, to wrap up this part of the discussion, TextPro-SLM is showing that aligning speech input with text-style representations while keeping the prosodic information intact is a very promising direction for building more capable and practical SLMs <ref:two thousand six hundred five point zero five nine two seven#pg5.
Lu: It’s data efficient because they only needed about one thousand hours of audio training, and the gains on reasoning tasks are significant when you compare them to other approaches like SALAD <ref:two thousand six hundred five point zero five nine two seven#pg5.
Meng: I think the implication for us is that if we can reduce the data requirements for these kinds of models while maintaining high performance on complex tasks, it opens up possibilities for much more accessible speech AI systems <ref:two thousand six hundred five point zero five nine two seven#pg5.
Lalam: The results suggest that aligning speech inputs with text-style representations, with prosodic information preserved, is a promising and data-efficient direction for building more capable and practical SLMs <ref:two thousand six hundred five point zero five nine two seven#pg5.
Tom: That’s the gist of what we saw here about TextPro-SLM. We need to keep an eye on how this dual representation approach scales when we move past these initial benchmarks <ref:two thousand six hundred five point zero five nine two seven#pg1.
Jane: Exactly. It’s a solid piece of work that shows how focusing on the input side, specifically by separating content from style, can lead to real gains in both reasoning and understanding speech cues <ref:two thousand six hundred five point zero five nine two seven#pg5.
Lu: We'll keep watching how they address those computational costs for deployment, because proving efficiency is just as important as proving capability <ref:two thousand six hundred five point zero five nine two seven#pg1.
Meng: Yeah, the engineering side of things is always the next hurdle after a great paper like this <ref:two thousand six hundred five point zero five nine two seven#pg1.
Lalam: So, to wrap up on this discussion about TextPro-SLM, it shows that aligning speech inputs with text-style representations while keeping the prosodic information intact is a promising and data-efficient direction for building more capable and practical SLMs <ref:two thousand six hundred five point zero five nine two seven#pg5.
The paper's summary: Tom: So, we've been talking about how speech models struggle to connect what they hear to what they actually understand, and this paper introduces TextPro-SLM as a way to fix that by changing how the speech gets into the model.
Jane: Right. Basically, the paper argues that the real problem isn't just making the model output sound better for text anymore; it’s about making sure it understands both what is being said and exactly how it’s being said when you feed it audio.
Tom: Exactly. They propose this TextPro-SLM architecture to make spoken input look more like what a text model expects, but they keep the prosodic cues woven in too.
Jane: So, they're not just trying to get the words right; they're explicitly separating and preserving those extra parts of speech that tell you things like the speaker’s emotion or their speaking style.
Tom: That sounds like a two-pronged approach. They use a Speech encoder, WhisperPro, which spits out both text tokens for what was said and prosody embeddings for how it was said.
Jane: And then the main language model backbone has to learn to handle both of those streams at the same time. It’s trained with two objectives working together.
Tom: They use a loss function that combines an auto-speech recognition loss with a mel reconstruction loss, setting the weight for each objective to one point zero. That forces it to pay attention to both the words and the sound simultaneously.
Jane: And what they found is that this method achieves the lowest gap between speech models and text models compared to other leading speech LLMs at both three billion and seven billion parameter sizes.
Tom: The numbers are pretty compelling, Jane. They achieved this with only about one thousand hours of training audio, which is way less than some previous estimates we've seen for this kind of performance level.
Jane: That’s the data efficiency angle there. It suggests that aligning speech input with text-style representations while keeping the prosodic information intact is a very promising direction for building more capable and practical speech AI systems.
Tom: And it's not just about the gap reduction; they also show that this model performs much better on reasoning tasks than some other models we've seen, like SALAD.
Jane: That’s important because it shows that improving how you handle the input side directly helps with complex thinking, not just simple word recognition.
Tom: So, to put it simply, TextPro-SLM suggests that if you can get the model to understand both the words and the music of speech at once from the start, you get a much better overall system for understanding spoken language.
Jane: It moves us away from just treating speech as a weird translation problem and toward treating it as a richer signal that has structure we need to preserve.
Tom: We’ll keep an eye on how they handle those computational costs when you try to run this dual encoder setup at scale, because that's always the next big hurdle for deployment.
The paper's improvements: Tom: So, we've talked about how TextPro-SLM makes the input look more like text by using that dual representation of words and prosody embeddings. And now we're going to talk about what this actually *does* for the model's capabilities.
Jane: Right. The paper highlights a few specific improvements they made in their architecture, and it really shows how those changes translate into better performance on different kinds of tasks.
Tom: One big improvement is that the way they train the language model backbone is much smarter now because it has to preserve the original text model's semantic knowledge while learning to understand those paralinguistic cues.
Jane: That means even when it’s focusing on understanding emotion or speaking style, it doesn't forget how to do its core reasoning tasks, which is something we always worry about in these speech models.
Tom: Exactly. They also showed this system can do strong paralinguistic understanding tasks like gender recognition and age prediction by training the backbone on specific datasets like IEMOCAP and ParaSpeechCaps.
Jane: That's interesting because it implies that even with just a single prosody embedding, which is what they used, the model can already hit state-of-the-art performance in understanding *how* things are spoken.
Tom: And for reasoning tasks specifically, the paper points out that this approach substantially reduces the gap when compared to models like SALAD on things like high school math.
Jane: The main implication here is that reducing the gap from the input side is actually quite effective, especially when you look at how much better TextPro-SLM performs on those heavy reasoning benchmarks.
Tom: They show this model achieving a modality gap of just one point eight percent on high school math compared to ten point five percent for other models, which is a significant reduction.
Jane: So what this means for someone listening? It means if you use an AI system that relies on spoken input, it's less likely to fail when the input isn't perfectly clean or text-like because the model can handle that noise better.
Tom: It shifts the focus from just making the output sound good to making sure the model has a solid foundation for both content and delivery simultaneously.
Jane: And while they show these wins, it’s worth remembering that they admit this method doesn't cover every single aspect of speech understanding yet, which is a fair limitation to keep in mind.
Tom: Right. So we see these tangible gains on data efficiency and reasoning power here, but we still need to figure out how much computational overhead this dual encoder adds when you try to deploy it in a real application.
Conclusion: Tom: So we’re wrapping up on this paper, "Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM". We’ve talked about how they use that dual representation to make speech inputs look more like text and how it helps with reasoning.
Jane: Right. The main implication is that we can build speech AI systems that are much better at understanding the nuances of human speaking, not just the words themselves.
Tom: I think it shows that focusing on getting the input representation right is a really efficient way to boost performance without needing massive amounts of new audio data.
Jane: Exactly. It suggests that aligning speech with text-style representations while keeping those prosodic details preserved is a promising direction for building more capable and practical speech AI systems.
Tom: And they did show this works across different sizes, both three billion and seven billion parameter scales, which is good for showing how robust the method is.
Jane: It’s a solid piece of work that shows how focusing on the input side, specifically by separating content from style, can lead to real gains in both reasoning and understanding speech cues.
Tom: I'm just going to say we need to keep an eye on how they handle those computational costs for deployment because proving efficiency is just as important as proving capability.
Jane: True. We need to see if this dual encoder setup adds too much overhead when you try to run it in a real-world application, because that’s the next big hurdle.
Lu: If this approach proves scalable, it fundamentally changes how we design input layers for any multimodal model that handles spoken language, moving away from just treating audio as raw data.
Meng: From an engineering standpoint, I’m still wondering about the complexity of maintaining those two synchronized streams during inference when you're trying to keep things fast and low latency for production use.
Lalam: I’m confident in my assessment; based on the results they shared, TextPro-SLM is one of the most data-efficient ways we’ve seen to bridge that gap between text and speech representations.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI