Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

arXiv:2606.09366 · cs.CL, eess.AS · Submitted 2026-06-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs".

Jane: The gist The Convex Gate (C-Gate) proposes a speech–LLM bridge that constrains all speech representations to lie within the LLM’s input embedding manifold with an architectural convex-hull constraint,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: The title itself suggests a big question about necessity, doesn't it? Is text the absolute minimum information required for speech understanding when you use an LLM? Tom Yeah, that’s what they are probing. They want to see if we can build a bridge between audio and the frozen LLM where text is the main guide but not necessarily the only thing that matters. Lu The authors are tackling this by proposing something called Convex Gate, which is a way to keep all speech representations inside the LLM’s input embedding space using an architectural convex-hull constraint.

Meng: So, instead of just picking tokens or learning totally free embeddings, they are imposing a geometric rule on how the speech gets translated into the LLM's vocabulary. Jane That sounds like a very structured way to handle the connection between continuous audio and discrete text tokens. Tom Right. It’s about controlling where those speech features land in the LLM’s massive embedding table, which is crucial when you want both accurate transcription and emotion recognition working at the same time.

The paper's summary: Tom: Basically, the core idea of Convex Gate is that every single frame from a speech input gets represented as a convex combination of the LLM’s token embeddings. Jane That means each piece of audio isn't just mapped to one word; it’s mapped to a mix of several words, which keeps it compatible with the pretrained LLM. Lu This constraint is what they call ensuring compatibility with the pretrained LLM while preserving continuous expressivity, which is key because you don't want to collapse everything into just discrete tokens.

Meng: That means even though the final output uses those discrete tokens, the way we get there—the bridge itself—is smooth and continuous. Jane So they are trying to avoid those two extremes: one that locks everything down too tightly to just words, and another that lets the representation drift away from where the LLM was trained. Tom That’s what Page two is talking about, where they describe how existing methods either collapse towards near one-hot distributions for ASR or let representations drift off the input manifold for continuous learning <ref:2606.09366#pg3>.

The paper's improvements: Jane: Now, they suggest a few ways to improve this approach, which is what the paper calls enhancements. One major suggestion is focusing on how the information flows through time rather than just looking at static token choices. Lu They point out that causal interventions show that the working channel is actually a time-ordered bridge trajectory through the trained LLM input manifold, not just a set of selected tokens at each moment.

Tom: That’s interesting because it shifts the focus from discrete retrieval to this continuous path through the embedding space. Meng And they showed that shuffling that bridge trajectory in time really hurt emotion recognition, reducing it to only twenty-two percent, and boosting ASR error massively, to two hundred forty-four point five percent. Jane So what this means is that the LLM needs an ordered sequence of embeddings for the speech, not just a collection of relevant tokens at every single frame.

Lu: And another point they make is that you can treat self-attention adaptation as insufficient if you don't have this trained embedding manifold constraint in place. Tom So it’s not just about tweaking the attention layers; it needs this fundamental geometric alignment with the LLM’s existing structure to work well.

Conclusion: Jane: So, to wrap up, they argue that for speech and LLMs working together, we need to constrain the geometry of that bridge because the useful signal is a continuous trajectory through that embedding space. Tom It seems like their main finding is that this controlled regime lets the frozen LLM consume speech in a way where ASR improves by up to forty-eight point seven percent relative under matched data, and emotion accuracy gets improved or preserved <ref:2606.09366#pg1,by up to 48.7% relative>. Lu The implication here is that we should constrain the bridge geometry rather than just focusing on the routing posterior because that's where the useful signal resides, as they put it.

Meng: From a deployment standpoint, this suggests that for production systems, you need to ensure that whatever mechanism you use to feed speech into your LLM respects this manifold constraint or you're going to run into instability. Jane It really boils down to making sure the speech input respects the LLM’s internal structure so it doesn't drift off course. Tom Yeah, so we see a path where we get both strong ASR and good emotion results by making that structural constraint explicit in the design of the bridge.

Lalam: I am confident in this direction because my literature review shows that aligning the geometry with the trained embedding table is what makes these interfaces robust. Lu It confirms what I was leaning towards about focusing on that ordered trajectory, which is a big piece of evidence for their claim. Jane So, to summarize "Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs," they show that by enforcing this convex-hull constraint, we can achieve strong joint performance across ASR and emotion recognition because the bridge stays in the right place.

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Microsoft Research Asia

cs.CL, eess.AS

Submitted: 2026-06-08

Updated: 2026-10-14

Comments: NeurIPS 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 87/100

The gist: The gist The Convex Gate (C-Gate) proposes a speech–LLM bridge that constrains all speech representations to lie within the LLM’s input embedding manifold with an architectural convex-hull

Key concepts

Convex Gate (C-Gate): C-Gate is a speech-LLM bridge that represents each audio frame as a convex combination of fixed token embeddings from the LLM. This architectural constraint ensures all speech representations stay within the LLM's input embedding space, preventing drift while allowing continuous mixtures learned from task supervision.
Convex Hull Constraint: This constraint forces speech representations to exist within the convex hull defined by the frozen token embeddings of the LLM. This mechanism fixes the output codebook structure while enabling smooth, continuous variations needed for different speech tasks, ensuring stability and interpretability.
Basis Drift Prevention: Constraining representations to lie within a fixed convex set prevents 'basis drift,' which is when new representation directions are introduced outside the LLM's original training distribution. This keeps the model grounded in its learned knowledge while still allowing for continuous speech expression.
Working Channel Trajectory: Causal interventions reveal that performance relies on a specific, time-ordered trajectory through the LLM's input manifold, not just a set of selected tokens. The working channel is this continuous path, which the convex-hull constraint is designed to preserve for effective speech processing.

Terminology

Alignment — 2× in this paper
Accuracy — 1× in this paper · explained in 1 episode
Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
Embeddings — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes
Multimodal Integration — 1× in this paper · explained in 2 episodes

Summary

The gist The Convex Gate (C-Gate) proposes a speech–LLM bridge that constrains all speech representations to lie within the LLM’s input embedding manifold with an architectural convex-hull constraint, achieving strong joint performance across automatic speech recognition (ASR) and emotion recognition.

How it works

The core idea of C-Gate is to represent each frame as a convex combination of token embeddings, ensuring compatibility with the pretrained LLM while preserving continuous expressivity Each frame is represented as a convex combination of token embeddings, ensuring compatibility with the pretrained LLM while preserving continuous expressivity. This ensures that all representations lie within the convex hull of the embedding table, preventing basis drift while preserving continuous expressivity.

The mechanism involves a three-step process for generating a speech bridge vector

Compute similarity between speech features and all LLM tokens
Select a small subset of relevant tokens
Represent the speech frame as a weighted combination of these tokens.

The final speech bridge vector is constructed using a deterministic top-16 support combination after full-vocabulary scoring, which is still in the word embedding manifold, with neither value projection nor post-codebook MLP applied. The normalized coefficient alpha t v weights embedding row Ev and the speech bridge vector inserted into the LLM M is the renormalized mixture of the top-K supports St.

Key Advantages of Geometric Constraint

Constraining representations to lie in convex(E) has three key advantages

(i) It prevents basis drift by disallowing the introduction of new representation directions outside the LLM’s training distribution.

(ii) It preserves continuous expressivity, allowing representations to vary smoothly rather than collapsing to discrete tokens.

(iii) It maintains interpretability, as the resulting representations remain directly analyzable using standard LLM probing and visualization tools.

Performance Gains Across Tasks

Under a setting of 960h LibriSpeech ASR data, ∼ 47h public emotion data, and 707M trainable parameters, C-Gate establishes a controlled ASR–paralinguistic transfer regime using a Whisperlarge-v3 encoder and a Qwen2.5-7B-Instruct LLM. The dual-task (ASR+Emotion) model C-Gate-2T improves LibriSpeech autoregressive WER from 7.76% to 4.78%, a 38.4% relative reduction, while reaching 97.1% RAVDESS out-of-distribution closed-set emotion completion, 0.9pp above a same-structure emotion-only baseline. The three-task (ASR+Emotion+Reasoning) stress-test model C-Gate-3T with dynamic reweighting loss further reduces ASR to 3.98% WER, a 48.7% relative reduction over single-task, at a emotion cost of 6.6pp.

Mechanistic Understanding

Causal interventions confirm that both the trajectory structure and alignment to the pretrained embedding manifold are critical for performance. Three causal interventions identify the working channel as the time-ordered bridge trajectory through the trained LLM input manifold, precisely the channel that the convex-hull constraint is designed to preserve. These interventions isolate three necessary factors: replacing waveform with zeros or RMS-matched Gaussian noise collapses emotion from 91.5% to 12.0%/10.0% and inflates ASR WER from 2.8% to 97.5%/98.5%, confirming that real acoustic content drives the reported numbers. Shuffling the bridge trajectory in time reduces emotion to 22.0% and inflates WER to 244.5%, establishing that the LLM consumes an ordered trajectory rather than the multiset of selected supports. Replacing the trained LLM embedding table with a same-shape Gaussian or row-permuted table erases both gains, isolating manifold alignment with the trained E as the load-bearing factor and eliminating codebook size or self-attention capacity as alternative explanations.

Conclusion and Implications

C-Gate identifies a controlled regime in which a frozen 7B-parameter LLM consumes speech through its own embedding manifold, with ASR improving by 48.7% relative under matched data and budget, emotion preserved or improved under C-Gate-2T, and spoken-reasoning probes treated as boundary measurements. The causal interventions of §5.4 support three design implications: constrain the bridge geometry rather than the routing posterior because the useful signal is a continuous trajectory through convex(E), treat self-attention adaptation as insufficient without the trained embedding manifold, and audit audio grounding before reporting reasoning gains rather than treating MCQ improvements as direct evidence of grounded spoken reasoning. The strongest open alternative is that C-Gate succeeds because it adds trainable capacity near the LLM rather than because the capacity is organized as a convex codebook query.

Table 1 shows that joint training improves ASR rather than merely preserving it, and under identical ASR data and trainable-parameter budget, C-Gate-2T reduces autoregressive WER from 7.76% to 4.78%, a 38.4% relative gain over C-Gate-ASR, while also improving RAVDESS emotion accuracy from 96.2% to 97.1%. Table 2 places C-Gate-3T alongside academic, public-data, and large-scale audio LLMs on LibriSpeech, MMSU, and MMAU. Causal interventions isolate three necessary factors. Table 3 reports the entropy and KL of the renormalized posterior π˜t,16 over that support forwarded to the LLM, with Hmax = log 16. The working channel is therefore the selected-support trajectory through the trained embedding manifold, not per-frame symbolic retrieval.

Table 3 shows that C-Gate-Emotion Emotion 2.737 0.987 0.0356 13.35 nats ≈ 19.3 bits. Causal interventions isolate three necessary factors. The working channel is therefore the selected-support trajectory through the trained embedding manifold, not per-frame symbolic retrieval. Table 3 reports that the entropy and KL of the renormalized posterior π˜t,16 over that support forwarded to the LLM, with Hmax = log 16. The working channel is therefore the selected-support trajectory through the trained embedding manifold, not per-frame symbolic retrieval. Table 3 reports that the entropy and KL of the renormalized posterior π˜t,16 over that support forwarded to the LLM, with Hmax = log 16. The working channel is therefore the selected-support trajectory through the trained embedding manifold, not per-frame symbolic retrieval. Table 3 reports that the entropy and KL of the renormalized posterior π˜t,16 over that support forwarded to the LLM, with Hmax = log 16. The working channel is therefore the selected-support trajectory through the trained embedding manifold, not per-frame symbolic retrieval. Table 3 reports that the entropy and KL of the renormalized posterior π˜t,16 over that support forwarded to the LLM, with Hmax = log 16.

Improvements for AI systems

Bold header: Geometric Constraint Implementation

The system can be improved by enforcing an architectural convex-hull constraint, ensuring that each frame is represented as a convex combination of token embeddings, ensuring compatibility with the pretrained LLM while preserving continuous expressivity. This prevents representation drift by restricting representations to lie within the convex hull of the embedding table, which is crucial for stability during autoregressive decoding.

Bold header: Improved ASR and Paralinguistic Transfer

The improved system can achieve a controlled ASR–paralinguistic transfer regime, as evidenced by C-Gate-3T improving LibriSpeech WER by up to 48.7% relative while matching or exceeding single-task emotion accuracy. This allows the LLM to handle transcription, prosody, and emotion simultaneously within a single decoder.

Bold header: Task-Conditional Readout

The system can better serve both modalities by allowing the frozen LLM to read the same representation differently based on task prompts; for instance, emotion prompts induce a pooled readout, whereas ASR prompts induce a position-selective readout. This enables the bridge to function as both an utterance-level emotion classifier and a position-sensitive ASR mechanism.

Bold header: Mechanistic Understanding of Information Flow

The system's performance can be optimized by focusing on the time-resolved trajectories in the embedding space rather than discrete token identities, confirming that the trajectory structure and alignment to the pretrained embedding manifold are critical for performance. This insight guides future design toward preserving this continuous trajectory.

Bold header: Robustness Against Input Perturbations

The system can be made more robust by understanding that shuffling the bridge trajectory in time reduces emotion to 22.0% and inflates WER to 244.5%, confirming the working channel is the time-ordered bridge trajectory rather than the unordered support set. This mandates an ordered sequence for speech processing, which is essential for accurate ASR and emotion transfer.