Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
The paper introduces Text as a Universal Information Bottleneck for Speech LLMs to address how to integrate continuous acoustic signals into pretrained models. It proposes Convex Gate (C-Gate), a bridge that constrains speech representations to lie within the LLM's input embedding manifold using convex combinations of frozen token embeddings. This allows the model to perform joint tasks like ASR and emotion recognition while maintaining continuous speech variation.