Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap
The research investigates why end-to-end speech LLMs perform worse than text models even when they match prompts. It analyzes speech processing in four open-weight models through three phases: structural transformation, semantic smearing, and decision instability. The gap is caused by 'Information Dilution,' where redundant speech frames spread semantic signal too thinly, preventing stable late-stage decisions.