FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation
eess.AS, cs.AI, cs.SD
Submitted: 2026-01-08
Updated: 2026-08-31
Comments: Accepted to Findings of EMNLP 2026, page 2
Code: https://github.com/myshell-ai/MeloTTS
License: http://creativecommons.org/licenses/by/4.0/
The gist: Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens.
Terminology
Abstract
Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens. Existing speech-language models project high-frame-rate acoustic features directly into the LLM input space, making long-context processing computationally prohibitive. Unlike images or videos, speech lacks spatial redundancy, making extreme token compression particularly challenging. To address this limitation, we propose FastSLM, a token-efficient architecture featuring the Hierarchical Temporal Abstractor (HTA), which progressively distills acoustic features across multiple temporal scales. HTA achieves an extreme compression rate of 1.67 tokens per second (97% reduction) while preserving essential linguistic information for downstream speech-language understanding. Experimental results demonstrate that FastSLM achieves competitive performance across diverse speech-language tasks while requiring substantially fewer speech tokens and FLOPs than existing speech-language models. The source code and model checkpoints are available at https://github.com/Lee-junseok1025/FastSLM.
Sources
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- GPT-4 Technical Report
- Common Voice: A Massively-Multilingual Speech Corpus
- Multilingual and Cross-Lingual Intent Detection from Spoken Data
- Qwen2-Audio Technical Report
- Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech
- Kimi-Audio Technical Report
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
- The Llama 3 Herd of Models
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Gaussian Error Linear Units (GELUs)
- Long-Form Speech Generation with Spoken Language Models
- A Call for Clarity in Reporting BLEU Scores
- KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehension
- Voxtral
- Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
- Mixed Precision Training
Related papers
- Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
- X-VC: Zero-shot Streaming Voice Conversion in Codec Space
- Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
- VM-ARRAYDPS: Virtual Microphone Augmented Diffusion Posterior Sampling for Unsupervised Blind Speech Separation
- Conditional Flow Matching for Visually-Guided Acoustic Highlighting
- Anonymization, Not Elimination: Utility-Preserved Speech Anonymization