SAM: A Mamba-2 State-Space Audio-Language Model

arXiv:2509.15680 · cs.SD, eess.AS · Submitted 2025-09-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "SAM: A Mamba-2 State-Space Audio-Language Model".

Jane: Here's the gist of the paper. SAM presents State-space Audio-language Model that integrates an audio encoder with a Mamba-2 backbone and achieves competitive results against larger transformer-based ALMs with fewer parameters SAM2.7B achieves 21.1 mAP on AudioSet and 17.6 SPICE on AudioCaps, matching or surpassing larger 7B transformer-based models with fewer parameters.

Tom: Model Architecture The model follows a standard multimodal LLM architecture comprising an audio encoder fa, a text encoder ft, a connector fc, and a Mamba-2 LLM Given a mel-spectrogram x, the audio encoder produces audio tokens Ha = fa(x), which are projected by the connector into audio embeddings Ea = fc(Ha).

Jane: The text encoder maps the prompt p and caption c to embeddings Ep = ft(p) and Ec = ft(c), respectively.

Tom: During training, the LLM uses the concatenated sequence

Ea, Ep, Ec: , while at inference it uses only

Ea, Ep: .

Paper summary: Jane: Moving on to the next part of this paper, which is just summarizing the main points and what it all means for audio language modeling in general.

Tom: Alright, so we're talking about "SAM: A Mamba-two State-Space Audio-Language Model <ref:2509.15680#pg1,SAM: A Mamba-2 State-Space Audio-Language Model>." The core thesis is that they’ve integrated an audio encoder with a Mamba two backbone to create a state space model for audio language modeling.

Lu: The paper states that SAM2 point 7B achieves twenty-one point one mAP on AudioSet and seventeen point six SPICE on AudioCaps, which is competitive against larger transformer-based models when you compare them by parameter count <ref:2509.15680#pg3>.

Jane: So what does this actually mean for the field? It suggests that state space models can be strong backbones for audio language modeling even when you are trying to keep the model size manageable.

Meng: I'm wondering if this performance holds up when we move away from these specific benchmarks, like AudioSet and AudioCaps, because those datasets might not represent the complexity of real-world audio understanding.

Lalam: The authors also provided a systematic analysis of how state space models interact with audio encoder outputs, specifically looking at how joint audio encoder finetuning helps adapt the token representation rank and similarity across different SSM sizes.

Tom: That analysis shows that for smaller state space models, you need those fine-tuned audio representations to utilize their capacity effectively, which is a key design insight they found.

Jane: And beyond that, they showed that compact, information-rich audio token representations benefit more from structured approaches like using separator tokens than just relying on the model's inherent linear scaling with sequence length.

Lu: They essentially established practical design principles for state space models as strong backbones for audio language models by showing how to tune them and how to structure their inputs effectively.

Tom: So the takeaway is that you don't necessarily need massive parameter counts in an AI model if you use a Mamba two architecture combined with smart audio representation and supervision.

Conclusion: Jane: We're wrapping up our look at "SAM: A Mamba-two State-Space Audio-Language Model <ref:2509.15680#pg1,SAM: A Mamba-2 State-Space Audio-Language Model>." To summarize, the authors are presenting a state space audio language model that integrates an audio encoder with a Mamba two backbone.

Tom: The key results they highlight are SAM2 point 7B achieving twenty-one point one mAP on AudioSet and seventeen point six SPICE on AudioCaps, matching or surpassing larger transformer-based models with fewer parameters <ref:2509.15680#pg3>.

Lu: And beyond the numbers, they showed that joint audio encoder finetuning is essential for accuracy gains and for adapting the token representation rank when you use smaller state space models.

Meng: It seems like they’ve given us a blueprint for how to build these kinds of models efficiently by showing how to structure the audio input and what kind of supervision helps unlock reasoning.

Lalam: So in simple terms, this paper shows that Mamba two can be a strong, scalable backbone for audio language models when you combine it with proper training techniques and representation strategies.

Taehan Lee, Jaehan Jung, Hyukjun Lee

Sogang University

cs.SD, eess.AS

Submitted: 2025-09-19

Updated: 2026-06-10

Comments: 6 pages, Accepted to Interspeech 2026

DOI: 10.21437/Interspeech.2026-639

Code: https://github.com/sam-audio-language-model/sam

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: SAM presents State-space Audio-language Model that integrates an audio encoder with a Mamba-2 backbone and achieves competitive results against larger transformer-based ALMs with fewer parameters…

Key concepts

State Space Models (SSMs)
SSMs are a type of neural network that discretizes continuous time sequences into a state space. They are efficient because they can process long audio sequences in near linear time, unlike traditional models that slow down as sequence length increases.
Mamba-2 Backbone
Mamba is an SSM architecture known for its ability to handle long-range dependencies efficiently. Mamba-2 specifically improves this performance by using input-dependent SSM parameters, allowing it to adapt its processing based on the specific audio data being fed into it.
Audio Encoder Finetuning
Fine-tuning the audio encoder is crucial for SSMs. It helps the model adapt its reduced capacity to better integrate complex audio information. This adaptation results in more compact and similar token representations, which improves overall model performance.

Terminology

Summary

SAM presents State-space Audio-language Model that integrates an audio encoder with a Mamba-2 backbone and achieves competitive results against larger transformer-based ALMs with fewer parameters SAM2. p. 37B achieves 21. p. 01 mAP on AudioSet and 17. p. 06 SPICE on AudioCaps, matching or surpassing larger 7B transformer-based models with fewer parameters. p. 0

Model Architecture

The model follows a standard multimodal LLM architecture comprising an audio encoder fa, a text encoder ft, a connector fc, and a Mamba-2 LLM Given a mel-spectrogram x, the audio encoder produces audio tokens Ha = fa(x), which are projected by the connector into audio embeddings Ea = fc(Ha). The text encoder maps the prompt p and caption c to embeddings Ep = ft(p) and Ec = ft(c), respectively. p. 0 During training, the LLM uses the concatenated sequence [Ea, Ep, Ec], while at inference it uses only [Ea, Ep]. p. 0

SSM Implementation Details

Structured State Space Models (SSMs) in discrete time transform a 1D input sequence x ∈ R T 7→ y ∈ R T as in (1). The discretized parameters (A,¯ B¯) are obtained from the continuous-time parameters (A, B) and a step size ∆ ∈ R via the zero-order hold (ZOH) discretization rule in (2). p. 0 Mamba [8] improves language-modeling performance with input-dependent SSM parameters for xt ∈ R D: ∆(d)t = softplus(W(d)∆ xt + βt), Bt = WBxt, Ct = Wcxt. p. 0 Mamba also introduces hardware-efficient computation kernels based on parallel selective scan. p. 0 A Mamba block consists of a per-channel SSM, a 1D convolution, and a gated MLP. p. 0

Audio Encoder Finetuning Necessity

Joint audio-encoder finetuning is essential for SSMs. Beyond accuracy gains, the authors observe that the audio encoder produces lower rank and more similar token representations for smaller SSMs, suggesting adaptation to the SSM’s reduced capacity to integrate audio information. p. 0 This is analyzed by measuring the τ-effective rank of extracted audio tokens as a proxy for embedding space utilization. p. 2 Smaller SSMs are paired with more reduced token representations (higher token-to-token cosine similarity and lower τ-effective rank) under endto-end training. p. 2

Token Representation Strategy

SSMs benefit more from rich yet compact audio token representations than from simply exploiting their lineartime and memory scaling with respect to sequence length. The authors compare three novel connector designs for producing the audio embeddings Ea: (a) concatenation, (b) time major, and (c) frequency major. p. 1 They propose inserting separator token embeddings (“&&”) to mark boundaries between time steps or frequency bands to better preserve structural cues in the embedding sequence. p. 1

Reasoning Enhancement via Supervision

Structured Binary Question (BQ) and Multiple-Choice Question (MCQ) supervision unlocks SSMs’ audio reasoning capability. Incorporating instruction-following supervision substantially improves reasoning ability, boosting MMAU-Sound accuracy from 22. p. 08 to 56.8. This structured instruction-following supervision is particularly effective in strengthening audio reasoning ability across all model scales. p. 3

Conclusion

In this work, Mamba-2 exhibits strong audio-language modeling performance, achieving competitive results against larger transformer-based ALMs while using fewer parameters. Through extensive experiments and analysis of SSM-specific properties, the authors provide practical insights for training and designing SSM-based audio-language models. p. 3 The paper establishes practical design principles for SSMs as strong, scalable backbones for audio-language models. p. 0

Index Terms

Audio Language Models, State Space Models SAM: A Mamba-2 State-Space Audio-Language Model SAM is a Mamba-2 State-Space Audio-Language Model. The code is publicly available at https://github. p. 0com/sam-audio-language-model/sam. p. 0

References

[1] A Vaswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, A Gomez, L u Kaiser, and I Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, I Guyon, U V Luxburg S Bengio H Wallach R Fergus S Vishwanathan and R Garnett Eds., vol 30. Curran Associates Inc. p. 4 p. 4, 2017. [Online]. Available: https://proceedings. p. 4 p. 4 p. 4neurips.cc/paper files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper. p. 4pdf

[2] Y Gong, H Luo, A H Liu, L Karlinsky, and J R Glass, “Listen, think, and understand,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview. p. 4 p. 4 p. 4 p. 4 p. 4 p. 4 p. 4 p. 4 p. 4 p. 4 p. 4 p. 4net/forum?id=nBZBPXdJlC

[3] S Ghosh, S Kumar, A Seth, C K R Evuru U Tyagi S Sakshi O Nieto R Duraiswami and D Manocha, “GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y Al-Onaizan M Bansal and Y.-N Chen Eds. p. 4 p. 4 Miami, Florida, USA: Association for Computational Linguistics, Nov. p. 4 p. 4 2024, pp. 6288–6313. p. 4 [Online]. Available: https://aclanthology. p. 4 p. 4 p. 4org/2024.emnlp-main. p. 4361/

[4] S Ghosh, Z Kong, S Kumar, S Sakshi J Kim W Ping R Valle D Manocha and B Catanzaro, “Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities,” in Forty-second International Conference on Machine Learning, 2025. [Online]. Available: https://openreview.net/forum?id=xWu5qpDK6U

[5] S Ghosh, A Goel, J Kim, S Kumar, Z Kong S gil Lee C H Yang R Duraiswami D Manocha R Valle and B Catanzaro, “Audio flamingo 3: Advancing audio intelligence with fully open large audio language models,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=FjByDpDVIO

[6] J Xu et al., “Qwen3-omni technical report,” 2025. p. 4 [Online]. Available: https://arxiv. p. 4 p. 4 p. 4org/abs/2509. p. 417765

[7] S Deshmukh, S Dixit, R Singh, and B Raj, “Mellow: a small audio language model for reasoning,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=um4aiicz3L

[8] A Gu and T Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” in First Conference on Language Modeling, 2024. [Online]. Available: https://openreview.net/forum?id=tEYskw1VY2

[9] T Dao and A Gu, “Transformers are ssms: generalized models and efficient algorithms through structured state space duality,” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024.

[10] W Huang, J Pan, Z Wang, Y Liu Y Wang S Shen and J Hu, “Enhancing multimodal large language models with efficient feature alignment and processing using state space models,” Neurocomputing, vol 665 p 132152, 2026. p. 4 [Online]. Available: https://www. p. 4sciencedirect. p. 4com/science/article/pii/S0925231225028243

[11] Y Qiao, Z Yu, Z Zhao, S Chen M Sun L Guo Q Wu and J Liu, “VL-Mamba: Exploring state space models for multimodal learning,” in Proceedings of The 4th NeurIPS Efficient Natural Language and Speech Processing Workshop, ser. Proceedings of Machine Learning Research M Rezagholizadeh P Passban S Samiee V Partovi Nia Y Cheng Y Deng Q Liu and B Chen Eds. p. 4, vol 262. PMLR, 14 Dec 2024, pp. p. 4 102–113. [Online]. Available: https://proceedings.mlr.press/v262/qiao24a. p. 4html

[12] G Pantazopoulos, M Nikandrou, A Suglia O Lemon A Eshghi and A Suglia, “Shaking up VLMs: Comparing transformers and structured state space models for vision & language modeling,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y Al-Onaizan M Bansal and Y.-N Chen Eds. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 14 318–14 337. p. 4 [Online]. Available: https://aclanthology.org/2024.emnlp-main

[13] S Bhati, Y Gong, L Karlinsky H Kuehne R Feris and J Glass, “State-space large audio language models,” 2024. [Online]. Available: https://arxiv.org/abs/2411. p. 415685

[14] A Gu K Goel A Gupta and C Re, “On the parameterization ´ and initialization of diagonal state space models,” in Advances in Neural Information Processing Systems, S Koyejo S Mohamed A Agarwal D Belgrave K Cho and A Oh Eds., vol 35. Curran Associates Inc., 2022, pp. p. 4 35 971–35 983. p. 4 [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2022/file/e9a32fade47b906de908431991440f7c-Paper-Conference. p. 4pdf

[15] L Gao, S Biderman S Black L Golding T Hoppe C Foster J Phang H He A Thite N Nabeshima S Presser and C Leahy, “The pile: An 800gb dataset of diverse text for language modeling,” 2020. [Online]. Available: https://arxiv.org/abs/2101. p. 400027

[16] W Chen Y Liang Z Ma Z Zheng and X Chen, “Eat: self-supervised pre-training with efficient audio transformer,” in Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, ser IJCAI ’24, 2024. [Online]. Available: https://doi. p. 4 p. 4 p. 4org/10.24963/ijcai. p. 42024/421

[17] J F Gemmeke D P W Ellis D Freedman A Jansen W Lawrence R C Moore M Plakal and M Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 776–780.

[18] W Chen et al. p. 4, “Slam-aac: Enhancing audio captioning with paraphrasing augmentation and clap-refine through llms,” in ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. p. 4 1–5.

[19] W Chen et al. p. 4, “Sjtu-thu automated audio captioning system for dcase 2024,” DCASE2024 Challenge, Tech. p. 4 Rep. 2, May 2024. p. 4

[20] Y Liu, Y Tian, Y Zhao H Yu L Xie Y Wang Q Ye J Jiao and Y Liu, “VMamba: Visual state space model,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. p. 4 [Online]. Available: https://openreview.net/forum?id=ZgtLQQR1K7

[21] E J Hu yelong shen P Wallis Z Allen-Zhu Y Li S Wang L Wang and W Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

[22] M Yoshimura T Hayashi and Y Maeda, “MambaPEFT: Exploring parameter-efficient fine-tuning for mamba,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=UAKnJMIBwf

[23] K Galim W Kang Y Zeng H I Koo and K Lee, “Parameterefficient fine-tuning of state space models,” in Forty-second International Conference on Machine Learning, 2025. [Online]. Available: https://openreview.net/forum?id=64mHSb9DlQ

[24] T Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=mZn2Xyh9Ec

[25] K J Piczak, “Esc: Dataset for environmental sound classification,” p. 1015–1018, 2015. p. 5 [Online]. Available: https://doi.org/10.1145/2733373. p. 52806390

[26] A Mesaros, A Diment B Elizalde T Heittola E Vincent B Raj and T Virtanen, “Sound event detection in the DCASE 2017 challenge,” IEEE/ACM Transactions on Audio, Speech and Language Processing, 2019, in press.

[27] Y Gong J Yu and J Glass, “Vocalsound: A dataset for improving human vocal sounds recognition,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. p. 5 151–155.

[30] H Chen W Xie A Vedaldi and A Zisserman, “Vggsound: A large-scale audio-visual dataset,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. p. 5 721–725.

[31] E Fonseca X Favory J Pons F Font and X Serra, “Fsd50k: An open dataset of human-labeled sound events,” IEEE/ACM Transactions on Audio, Speech and Language Processing, vol 30, pp. p. 5 829–852, 2022. p. 5

[32] P Anderson B Fernando M Johnson and S Gould, “Spice: Semantic propositional image caption evaluation,” in Computer Vision – ECCV 2016, B Leibe J Matas N Sebe and M Welling Eds. p. 5 Cham: Springer International Publishing, 2016, pp. p. 5 382–398.

[33] C D Kim B Kim H Lee and G Kim, “AudioCaps: Generating captions for audios in the wild,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J Burstein C Doran and T Solorio Eds. p. 5 Minneapolis, Minnesota: Association for Computational Linguistics, Jun. p. 5 2019, pp. 119–132. [Online]. Available: https://aclanthology.org/N19-1011/

[34] K Drossos S Lipping and T Virtanen, “Clotho: an audio captioning dataset,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 736–740.

[35] B Elizalde S Deshmukh M A Ismail and H Wang, “Clap learning audio concepts from natural language supervision,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. p. 5 1–5.

[36] J Li D Li S Savarese and S Hoi, “Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models,” in Proceedings of the 40th International Conference on Machine Learning, ser ICML’23. p. 5 JMLR.org, 2023.

[37] H Wang M Mendiratta C Theobalt and A Kortylewski, “Facegpt: Self-supervised learning to chat about 3d human faces,” CoRR, vol abs/2406. p. 507163, 2024. p. 5 [Online]. Available: https://doi.org/10.48550/arXiv. p. 52406.07163

[38] Y Meng J Krishnan S Wang Q Wang Y Mao H Fang M Ghazvininejad J Han and L Zettlemoyer, “Representation deficiency in masked language modeling,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=b3l0piOrGU

[39] G Team, “Gemma 3n,” 2025. [Online]. Available: https://ai. p. 0google.dev/gemma/docs/gemma-3n

[40] S Sakshi U Tyagi S Kumar A Seth R Selvakumar O Nieto R Duraiswami S Ghosh and D Manocha, “MMAU: A massive multi-task audio understanding and reasoning benchmark,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Audio Language Model Backbone Efficiency: Replace quadratic attention mechanisms in transformer backbones with Mamba-2 SSMs to achieve competitive performance with fewer parameters, such as SAM-2.7B matching or surpassing 7B transformer models on AudioSet and AudioCaps.

  2. Joint Encoder Finetuning for SSMs: Implement joint audio encoder finetuning because it is essential for SSMs, leading to observed accuracy gains and adaptation of token representation rank and similarity across different SSM sizes.

  3. Optimized Token Representation Strategy: Utilize compact, information-rich audio token representations instead of simply scaling sequence length to benefit from the linear scaling properties of SSMs.

  4. Instruction Following Supervision for Reasoning: Incorporate structured Binary Question (BQ) and Multiple-Choice Question (MCQ) supervision to substantially improve reasoning ability, boosting MMAU-Sound accuracy from 22.8 to 56.8 (+34 points).

  5. Adaptive Multimodal Connector Design: Employ novel connector designs like time major or frequency major token rearrangement with separator tokens ("&&") to better preserve structural cues in the SSM state updates, resulting in sequences lengths of Ea = 576 for (b) and Ea = 520 for (c).

  6. Size-Matched Encoder Transfer: Train smaller models using a frozen audio encoder initialized from the converged checkpoints of a larger model, as size-matched encoder achieves the strongest performance in most cases.

  7. Reasoning Enhancement via OpenReasonAQA: Train Mamba-2 with intensive reasoning supervision derived from OpenReasonAQA, which replaces caption-style questions with structured BQ/MCQ reasoning questions to strengthen audio reasoning across all model scales.

Sources

Related papers