ALARM: Audio-Language Alignment for Reasoning Models

arXiv:2603.09556 · cs.CL · Submitted 2026-03-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "ALARM: Audio-Language Alignment for Reasoning Models".

Paper summary: Tom: So looking at the whole "ALARM: Audio–Language Alignment for Reasoning Models" paper by Grinberg and Shahmohammadi, they've shown a way to integrate audio understanding into large language models that have chain of thought reasoning capabilities.

Jane: It seems the core idea is using this self-rephrasing mechanism to create training targets that have natural thinking traces while keeping the original model aligned with what it was trained on.

Lu: They also tackled the input representation by fusing multiple audio encoders through cross-attention and Perceiver based techniques, which lets them get robust understanding of both vocal and non-vocal signals in a compressed format.

Meng: From an engineering standpoint, this framework allows their four billion parameter ALM to achieve top results on reasoning benchmarks like MMSU while requiring substantially lower training cost and amount of data than many other models.

Tom: The implication is that we can get strong audio understanding without having to fine-tune the main language model for every single audio task, which is a big thing for deployment.

Jane: It really shifts the focus from just making the LLM talk about audio to actually giving it a way to reason and produce output that makes sense when you hear it spoken.

Conclusion: Tom: So we're wrapping up on ALARM, which is about making large language models actually understand audio signals while they’re doing reasoning tasks.

Jane: Right, so the title itself, "Audio-Language Alignment for Reasoning Models," it tells us they aren't just trying to make the AI listen; they are trying to link what the AI understands about sound with its ability to think through a problem.

Lu: Exactly. The core idea is this self-rephrasing technique, which takes the model's own output and turns it into something that helps it reason while keeping its original training intact.

Meng: From my side, what I see is they managed to do this with a four billion parameter model, and the training cost for this specific approach is pretty low compared to other heavy audio models we've seen.

Tom: It sounds like they took a really complex setup—fusing Whisper, W2VBERT, MuQ, and SSLAM encoders—and managed to keep the main language model frozen while still getting better performance on speech benchmarks.

Jane: And that fusion part is key because it shows they don't have to rely on just one way of understanding sound; they're combining different audio tools in a clever way.

Lu: They used three different ways to fuse those audio encoders: stacking cross-attention modules, treating the Whisper features as the main thing with Perceiver modules, or concatenating adapted encodings.

Meng: I'm curious about deployment though; how much extra compute does adding all these encoders and fusion methods actually add when you try to run this on real traffic?

Tom: Well, they show that their feature-ensembling method, ALARM-E, is the strongest performer on the speech benchmark we're looking at right now.

Jane: And what it means for us is that we can get really high accuracy on tasks like reasoning and speech understanding without having to fine-tune the entire language model for every new audio thing.

Lu: That’s because they keep the reasoning part of the model frozen and just train those small audio adapters, which saves a ton of effort.

Meng: Still, I gotta ask about those limitations; they mentioned that fine-tuning a text-based LLM actually causes its original textual knowledge to degrade a bit.

Tom: That’s a fair caveat. It tells us that there's an inherent trade-off when you try to teach an existing language model something new, even if it's just audio understanding.

Jane: So the takeaway is that this framework gives us a way to build smarter AI that handles reasoning and sound together, using methods that are efficient enough for real world models.

Lu: And the next thing we need to watch is how they handle those compression techniques for those audio tokens, because if you can keep the token rate low while boosting performance, that's huge.

Petr Grinberg, Hassan Shahmohammadi

EPFL · Sony Europe Ltd.

cs.CL

Submitted: 2026-03-10

Updated: 2026-03-10

Comments: Submitted to Interspeech2026

Journal ref: Interspeech 2026 [Long Track]

DOI: 10.21437/Interspeech.2026-759

Code: https://github.com/Blinorot/ALARM

Project page: https://sakshi113.github.io/mmau_homepage/#leaderboard-v15-parsed

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

The gist: Large audio language models (ALMs) extend LLMs with auditory understanding, and this paper proposes self-rephrasing to convert self-generated responses into audio-understanding variants compatible…

Key concepts

Self-Rephrasing
This technique converts a model's generated response into a specific textual format that is compatible with audio understanding systems. It ensures the output retains the original meaning and reasoning structure of the LLM while adapting it for an audio context, preventing mismatches in how the model generates text versus how it should be interpreted as audio.
Audio Adapters
These are trainable components added to a frozen Reasoning Language Model (RLM) to allow it to interpret and produce outputs based on audio representations. By training only these adapters, the core reasoning capabilities of the LLM remain unchanged, making the adaptation efficient and cost-effective.
Feature Ensembling (ALARM-E)
This involves combining features from several different audio encoders—like Whisper for speech and W2VBERT for auditory cues. ALARM-E specifically concatenates adapted Whisper encodings with features from another fusion method, creating a robust representation of the audio signal that boosts performance significantly.

Terminology

Summary

Large audio language models (ALMs) extend LLMs with auditory understanding, and this paper proposes self-rephrasing to convert self-generated responses into audio-understanding variants compatible with reasoning LLMs while preserving distributional alignment The 4B ALM, ALARM-E, outperforms models of similar size and even surpasses most larger ALMs on MMSU [18], ranking third among the other models while preserving the backbone’s textual capabilities with a low training cost. p. 1

Pruning redundant reasoning tokens

The authors address the failure of standard self-generation techniques for state-of-the-art LLMs with built-in chain-of-thought reasoning capabilities (RLMs) because replacing Aaudio with Atext is insufficient for RLMs, as their built-in reasoning process reveals the textual nature of the input, resulting in unnatural responses during inference. To tackle this issue, they propose a self-rephrasing approach that converts generated responses R into textual audio-understanding variants while still avoiding output-distribution mismatch. p. 0 This rephrasing is applied to both the reasoning and the final-answer parts of R0, where Ri0 is sampled from Qr conditioned on (Aitext, Pi) and Ireph specifies the rephrasing rules. p. 2 The rephrased response Rtext is generated by the same frozen Qr, which avoids weight updates and mitigates output-distribution mismatch. p. 2

How the training signal is built

The dataset D is constructed to be D = (Ai audio, Pi, Ri text), where Pi is a prompt about Ai audio and Ri text is the corresponding textual response from an RLM. Constructing this dataset involves two main challenges: (1) Generating a diverse set of prompts P while ensuring consistency with the corresponding metadata Atext, and (2) The responses R should not reveal the textual nature of the input. p. 1 To address challenge (1), they use a fixed, pretrained instruction LLM Q as a prompt generator to sample candidate prompts P i from Q conditioned on Aitext, I, 1. p. 1 They then filter P i to retain only prompts that are answerable given the information in Aitext and do not explicitly expose the textual nature of the input. p. 1 To address challenge (2), they keep the RLM model Qr frozen and generate natural targets via a two-stage self-rephrasing procedure. p. 1

Where the gains come from

The ALARM model consists of a frozen RLM Qr augmented with trainable audio adapters and frozen encoders, enabling the backbone to interpret audio representations and produce desired outputs. They propose three fusion methods for stronger representations by integrating multiple audio encoders, operating at a 50 Hz token rate or lower through compression. p. 0 The ALARM-E approach achieves the highest open-source accuracy on MMAU-Speech and ranks top-3 among all the models, outperforming even Gemini-1. p. 55-Pro [63] and MiMo-Audio-7B-Instruct-Think. p. 5

Encoder Comparison and FineGrained Evaluation

The ALARM model employs a multi-encoder design using Whisper for speech, W2VBERT-2.0 for richer auditory cues, MuQ for music, and SSLAM for general sound understanding. p. 2 They propose three fusion methods: ALARM-CA stacks consecutive cross-attention fusion modules, ALARM-P treats Whisper’s features as the primary input with Perceiver modules for other encoders, and ALARM-E concatenates adapted Whisper encodings with h i3 from ALARM-CA along the time dimension. p. 2 The results show that while cross-attention fusion in ALARM-CA underperforms on speech category while improving on sound and music categories, ALARM-E combines the perceptual strengths of ALARM-CA with the content understanding of AL-Whisper-InstructR leading to substantial gains on MMAU-Speech. p. 5 The analysis confirms that while fine-tuning the text-based LLM improves performance on audio tasks, it leads to degradation of its pretrained textual knowledge. p. 7 The ALARM-E model demonstrates that medium-scale training on paired text–audio data is sufficient for strong speech understanding, even without fine-tuning the LLM. p. 7

General Audio Understanding

ALARM-E achieves the highest open-source accuracy on MMAU-Speech and ranks top-3 among all the models, including both proprietary and openaccess models according to its current official leaderboard. It surpasses DeSTA-2. p. 55-Audio [5], the previous opensource leader, by 5. p. 57% on the test-mini and 1. p. 58% on the full test speech subset. p. 5 Overall, ALARM-E outperforms most larger ALMs, including GPT-4o Audio [62], and Audio Flamingo 2 [80] on overall MMAU score. p. 5 The model keeps the RLM frozen and uses over 200× fewer audio tokens compared to other ALM variants, preserving its original text capabilities. p. 5 The results support the effectiveness of their fusion strategies that are able to boost the overall performance while keeping the token rate lower than all the encoders combined. p. 6

Conclusion

In this paper, ALARM is a framework for integrating audio understanding into large language models with chain-of-thought reasoning capabilities. Their self-rephrasing mechanism extends the self-generation paradigm to reasoning LLMs, producing training targets with natural thinking traces while preserving distributional alignment with the frozen backbone. p. 7 By removing the dependency on ASR-derived inputs and fusing multiple audio encoders through cross-attention and Perceiverbased techniques, their approach enables robust understanding of both vocal and non-vocal audio signals within compact feature representations. p. 7 Their feature-ensembling method (ALARM-E) allows their 4B-parameter ALM to achieve top-tier results on reasoning-oriented benchmarks, such as MMSU and speechMMAU, outperforming models of comparable and even larger scale while retaining the backbone’s textual capabilities at substantially lower training cost. p. 7 Their observations confirm that while fine-tuning the text-based LLM improves the performance on audio-related tasks, it leads to degradation of its pretrained textual knowledge. p. 7 The analysis of encoder compression and fusion reveals that while augmenting Whisper embeddings with complementary audio features is beneficial, the fusion mechanism must be meticulously designed to maintain robustness across diverse audio domains. p. 7

References

[1] H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. p. 7 Mian “A comprehensive overview of large language models” ACM Transactions on Intelligent Systems and Technology vol 16 no 5 pp 1–72 2025

[4] K.-H. Lu, Z. Chen, S.-W. Fu, H. Huang, B. Ginsburg, Y. p. 7-C. F. Wang, and H yi Lee “Desta: Enhancing speech language models through descriptive speech-text alignment” in Interspeech 2024 2024 pp 4159–4163

[5] K.-H. Lu, Z. Chen, S.-W. Fu, C.-H. H. Yang, S.-F Huang, C. p. 7-K Yang, C.-E Yu, C.-W Chen et al “Desta2 5-audio: Toward general-purpose large audio language model with self-generated cross-modal alignment” arXiv preprint arXiv:2507. p. 702768 10 Mar 2026

[13] K.-H. Lu, Z. Chen, S.-W. Fu, C.-H. H. Yang, J Balam, B Ginsburg Y-C F Wang and H-y Lee “Developing instructionfollowing speech language model without speech instructiontuning data” in ICASSP 2025-2025 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) IEEE 2025 pp 1–5

[14] R. Fan, B. Ren, Y. Hu, R. Zhao, S Liu, and J Li “Alignformer: Modality matching can achieve better zero-shot instructionfollowing speech-llm” IEEE Journal of Selected Topics in Signal Processing 2025

[15] D. Guo, D Yang, H Zhang, J Song, R Zhang, R Xu, Q Zhu et al “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning” arXiv preprint arXiv:2501. p. 712948 2025

[16] A Yang, A Li, B Yang, B Zhang, B Hui, B Zheng, B Yu et al “Qwen3 technical report” arXiv preprint arXiv:2505.09388 2025

[17] A Jaegle, S Borgeaud J-B Alayrac C Doersch C Ionescu D Ding S Koppula D Zoran A Brock E Shelhamer O Henaff M Botvinick A Zisserman O Vinyals and J Carreira “Perceiver IO: A general architecture for structured inputs & outputs” in International Conference on Learning Representations 2022

[18] D Wang, J Wu, J Li, D Yang, X Chen T Zhang and H Meng “Mmsu: A massive multi-task spoken language understanding and reasoning benchmark” arXiv preprint arXiv:2506.04779 2025

[19] S Sakshi U Tyagi S Kumar A Seth R Selvakumar O Nieto R Duraiswami S Ghosh and D Manocha “MMAU: A massive multi-task audio understanding and reasoning benchmark” in The Thirteenth International Conference on Learning Representations 2025

[20] A Nagrani J S Chung and A Zisserman “Voxceleb: A largescale speaker identification dataset” in Interspeech 2017 2017 pp 2616–2620

[38] K Lee K Park and D Kim “Dailytalk: Spoken dialogue dataset for conversational text-to-speech” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) IEEE 2023 pp 1–5

[47] K J Piczak “Esc: Dataset for environmental sound classification” in Proceedings of the 23rd ACM international conference on Multimedia 2015 pp 1015–1018

[48] E Fonseca X Favory J Pons F Font and X Serra “Fsd50k: an open dataset of human-labeled sound events” IEEE/ACM Transactions on Audio Speech and Language Processing vol 30 pp 829–852 2021

[49] J F Gemmeke D P Ellis D Freedman A Jansen W Lawrence R C Moore M Plakal and M Ritter “Audio set: An ontology and human-labeled dataset for audio events” in 2017 IEEE international conference on acoustics speech and signal processing (ICASSP) IEEE 2017 pp 776–780

[53] Y Chu J Xu Q Yang H Wei X Wei Z Guo Y Leng Y Lv J He J Lin et al “Qwen2-audio technical report” arXiv preprint arXiv:2407.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improved self-rephrasing mechanism that converts generated responses R into textual audio-understanding variants while avoiding output distribution mismatch by keeping the reasoning model Qr frozen during training and generating natural targets via a two-stage self-rephrasing procedure.

  2. Multi-encoder ALARM architecture that uses Whisper for speech, W2V-BERT-2.0 for richer auditory cues, MuQ for music, and SSLAM for general sound understanding to create robust representations from multiple audio encoders.

  3. ALARM fusion methods like ALARM-CA (stacking cross-attention blocks) or ALARM-P (using Perceiver modules) that effectively fuse features from multiple encoders to produce a final fused representation h i 3, enabling the model to produce the desired outputs when conditioned on this fused representation and instruction prompt.

  4. A compressed token sequence derived from multi-encoder inputs that achieves a lower token rate, such as 25 Hz or 50 Hz, compared to naive concatenation of all encoders (175 Hz), reducing computational cost and memory usage while maintaining strong performance.

  5. ALARM-E feature-ensembling strategy that concatenates the adapted Whisper encodings with h i 3 from ALARM-CA along the time dimension to form a 50 Hz frame rate, allowing the model to attend to distinct aspects of the audio signal through an auxiliary instruction Ie.

  6. The resulting ALARM-E model can achieve top-tier results on reasoning benchmarks like MMSU and speechMMAU, outperforming models of comparable or even larger scale while retaining the backbone’s textual capabilities at substantially lower training cost.

  7. ALARM systems can maintain full text performance by keeping the RLM frozen, demonstrating that multimodal fine-tuning drastically degrades the performance on purely textual tasks in contrast to ALARM models which do not modify the backbone weights.

Sources

Related papers