Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap

The gist The modality gap stems from an inherent redundancy in speech, which prevents stable decision-making in end-to-end speech LLMs.

How it works

1. The research investigates the dynamic roots of this modality gap beyond static geometric alignment by analyzing how speech and text representations evolve layer-by-layer on Page 1, five or more words copied word for word from that page: In this paper, we investigate the dynamic roots of this modality gap beyond static geometric alignment, analyzing how speech and text representations evolve layer-by-layer.

2. The authors evaluate four open-weight end-to-end models on SpeechMMLU and VoiceBench BBH on Page 1, five or more words copied word for word from that page: We evaluate four open-weight end-to-end models on SpeechMMLU and VoiceBench BBH.

3. The analysis identifies three distinct phases of speech input processing in LSLMs on Page 2, five or more words copied word for word from that page: We uncover three distinct phases of speech input processing in LSLMs:

4. Phase I is Structural Transformation, where speech features initially reside in a manifold distinct from text, necessitating a transformation phase before alignment can occur on Page 5, five or more words copied word for word from that page: "Phase I: Structural Transformation. We investigate whether speech adapter outputs initially reside in a manifold distinct from text embeddings, requiring early Transformer layers to perform a non-linear projection before alignment becomes possible."

5. Phase II is Semantic Smearing, confirmed by the finding that speech representations align closely with text in mid layers, yet often fail to stably form the correct late-layer decision on Page 6, five or more words copied word for word from that page: "Phase II: Semantic Smearing. We examine if speech’s inherent redundancy, where a single semantic unit spans multiple frames, manifests as distributed cross-layer alignment rather than sharp 1:1 token correspondence with text."

6. Phase III is Decision Instability, demonstrated by the observation that speech representations fail to stably isolate the correct lexical choice in late layers on Page 7, five or more words copied word for word from that page: Phase III: Decision Instability. Finally, we demonstrate that even after mid-layer alignment, speech representations fail to stably isolate the correct lexical choice in late layers.

7. The modality gap is attributed to a structural mismatch in semantic information granularity, specifically the contrast between dense text tokens and redundant speech frames on Page 2, five or more words copied word for word from that page: We identify the root cause of the modality gap as a structural mismatch in semantic information granularity, specifically the contrast between dense text tokens and redundant speech frames.

8. The mechanism is termed Information Dilution, stemming from the inherent redundancy of speech which leaves the per-token signal too weak to drive a reliable last-mile collapse on Page 2, five or more words copied word for word from that page: "We attribute this to a mechanism we term Information Dilution: stemming from the inherent redundancy of speech, semantic units are spread across multiple frames, leaving the per-token signal too weak to drive a reliable last-mile collapse."

9. The paper shows that simple statistical calibration is insufficient and can be detrimental when applied at the input layer on Page 1, five or more words copied word for word from that page: Crucially, simple statistical calibration is insufficient and can be detrimental when applied at the input layer.

10. The failure mode in models like Qwen2-Audio is termed Representation Failure, where speech representations never properly align with text geometry at any stage of processing on Page 17, five or more words copied word for word from that page: We term this pattern Representation Failure to distinguish it from the Phase III Decision Instability observed in better-performing models.

How it works

The paper employs a comprehensive anatomy of the mechanism across four open-weight LSLMs on Page 1, five or more words copied word for word from that page: We provide a comprehensive anatomy of this mechanism across four open-weight LSLMs.

The diagnostic framework uses three metrics to quantify cross-modal alignment: Cross-Modal Centered Kernel Alignment (CKA), Layerwise Token Norm, and Standardized Euclidean Distance on Page 3, five or more words copied word for word from that page: To rigorously evaluate the representational convergence between the speech and text modalities within the model, we employ a set of geometric and structural similarity metrics.

The analysis is structured into three phases to map the processing dynamics: Structural Transformation, Semantic Smearing, and Decision Instability on Page 4, five or more words copied word for word from that page: To analyze the modality gap, we structure our analysis into three phases, namely Structural Transformation, Semantic Smearing, and Decision Instability.

How it works

Phase I is characterized by early dark zone in the shallow layers where speech representations are not directly in the same feature geometry as text embeddings on Page 4, five or more words copied word for word from that page: In the early layers, speech representations have low similarity to any text layer, corresponding to the early dark zone shown in Figure 2. This indicates that early Transformer layers must perform a non-linear projection before alignment becomes possible on Page 6, five or more words copied word for word from that page: In the early layers, speech representations have low similarity to any text layer, corresponding to the early dark zone shown in Figure 2.

Phase II is characterized by a broad cross-layer alignment band arising from redundancy where semantic information spreads across multiple frames on Page 7, five or more words copied word for word from that page: "As shown in Figure 5, the layer-wise update vectors exhibit high cosine similarity between speech and text update vectors, indicating that the transformations applied at each layer are synchronized even when the cross-layer geometry exhibits a broad band."

Phase III is characterized by Decision Instability, where representations fail to complete final sharpening because no single token carries enough signal to drive a confident decision on Page 7, five or more words copied word for word from that page: "This is the signature of decision instability. Speech fails to separate the correct answer from strong competitors, because semantic information is spread across many redundant tokens and no single token carries enough signal to drive a confident decision."

How it works

The paper tests causal factors related to redundancy by performing controlled interventions on Page 12, five or more words copied word for word from that page: We perform a controlled intervention on Qwen2.5-Omni in the S2T setting.

One intervention involves injecting artificial redundancy into the T2T input by repeating each word in the question stem 1, 2, 4, or 8 times while keeping the answer options unchanged on Page 7, five or more words copied word for word from that page: Omni. We inject artificial redundancy into the T2T input by repeating each word in the question stem 1, 2, 4, or 8 times while keeping the answer options unchanged.

Another intervention is a KV token merging scheme applied to Qwen2.5-Omni where we compute pairwise cosine similarity among key vectors and merge tokens whose similarity exceeds a threshold of 0.90 on Page 13, five or more words copied word for word from that page: We implement a simple KV token merging scheme on Qwen2.5-Omni in the S2T setting.

How it works

The results of these interventions suggest that adaptive token merging or hierarchical pooling could compress distributed speech content into fewer, information-dense tokens on Page 19, five or more words copied word for word from that page: "This motivates exploring whether reducing redundancy at inference time can partially mitigate the speech modality gap.

1. Bold token merging at inference time to compress distributed speech content into fewer, information-dense tokens. This would allow models to make confident decisions as it does with text, mitigating Phase II (Semantic Smearing) and Phase III (Decision Instability) failures by reducing the signal dilution caused by redundant tokens.

2. Implement a mechanism to selectively retain task-relevant content instead of simply reducing token count, such as learned abstractors or hierarchical pooling. This addresses the finding that speech tokens retain acoustic residue that may interfere with late-layer sharpening, focusing on token quality over quantity.

3. Develop input layer projection methods that are non-linear rather than relying on simple statistical calibration (like mean and variance matching). This prevents the catastrophic performance collapse observed when forcing statistical alignment at the adapter output, thereby preserving the necessary structural transformation of speech features in early layers.

4. Integrate attention mechanisms designed to prioritize sharp semantic signals over diffuse ones, addressing the finding that speech attention is systematically more diffuse than text. This would enable the model to achieve late-layer sharpening by ensuring a single token carries enough signal to drive a confident decision, as observed in BBH cases.

5. Create task-aware decision margins where speech margins are actively trained to remain positive in late layers, mirroring the behavior of text. This directly targets the decision instability by forcing the model to separate the correct answer from strong competitors, addressing why speech margins often stay negative while text margins grow positive in late layers.