AQA-TTRL: Self-Adaptation in Audio Question Answering with Test-Time Reinforcement Learning

summary

The gist

Large Audio Language Models (LALMs) exhibit strong capabilities in general audio understanding but remain static after deployment…

In short

Large Audio Language Models (LALMs) are static after deployment, limiting their real-world adaptability. AQA-TTRL proposes a novel framework for audio understanding that allows models to self-improve on unlabeled test data using test-time reinforcement learning. It generates pseudo-labels via majority voting and uses Group Relative Policy Optimization (GRPO) guided by confidence weighting and multiple attempts to achieve stable, label-free adaptation.

Key concepts

Pseudo-label Generation
The model makes multiple predictions for a question, and a consensus pseudo-label is created by taking the majority vote among these outputs. This acts as a self-generated reward signal that guides the model's learning process without needing human labels.
Group Relative Policy Optimization (GRPO)
This is an optimization technique used to update the model's policy based on rewards derived from comparing its responses against the generated pseudo-labels. It helps in stabilizing updates by focusing on relative performance within groups of responses.
Confidence-Weighted Advantage
This mechanism adjusts the importance of each response's reward based on how confident the pseudo-label is. Higher confidence signals are given larger weights, ensuring that updates are guided primarily by reliable self-generated feedback.
Multiple-Attempt Sampling
This strategy addresses 'advantage collapse' by sequentially trying different response groups. If early attempts fail to produce diverse results, the system switches to the next group, ensuring that high-quality samples contribute effectively to learning.

Terminology used across episodes

This episode discusses

The paper

AQA-TTRL: Self-Adaptation in Audio Question Answering with Test-Time Reinforcement Learning · Read on arXiv

Haoyu Zhang, Jiaxian Guo, Dong Yang, Yusuke Iwasawa, Yutaka Matsuo

University of Tokyo

DOI: 10.21437/Interspeech.2026-288

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "AQA-TTRL: Self-Adaptation in Audio Question Answering with Test-Time Reinforcement Learning".

Dev: Here's the gist of the paper. Large Audio Language Models (LALMs) exhibit strong capabilities in general audio understanding but remain static after deployment, limiting their adaptability to real-world data Since supervised fine-tuning is costly, we propose AQA-TTRL, a novel framework for audio understanding that enables on-thefly evolution via test-time reinforcement learning using only unlabeled test data This paper introduces AQA-TTRL, a novel self-adaptation method that builds a closed-loop for selfimprovement (Fig.

Rosa: 1) The framework first establishes its consensus on an answer by performing majority voting over multiple outputs, creating a pseudo-label that acts as a self-generated reward signal This signal then guides policy optimization using Group Relative Policy Optimization (GRPO)

8, 9: The authors introduce two innovations to address the challenges of noisy pseudo-labels and advantage collapse These innovations are confidence-weighted advantage and multiple-attempt sampling.

Dev: Pruning redundant reasoning tokens The core mechanism of AQA-TTRL involves two main stages: pseudo-label generation and pseudo-label guided model updates For each audio-question pair (a, q), the model generates M predictions Y = 1, 2,..., yM It then conducts majority voting over candidate answers c ∈ Y to aggregate these stochastic predictions into a consensus pseudo-label: yˆ = arg max c∈Y PM m=1 1m=1 yˆ: pseudo-label For model updates, the authors adopt Group Relative Policy Optimization (GRPO)

8, 9: The policy model πθ generates a set of responses = o1, o2,..., oG for each audio-question pair Subsequently, the reward score ri is calculated by comparing the response oi with the pre-generated pseudolabel yˆ using the reward function The authors adopt two binary exact-match rewards: a format reward and an accuracy reward, defined as ri = racc(oi, yˆ) + rformat(oi) The advantage Ai is derived by normalizing the rewards within each group The policy model πθ is then optimized by maximizing the objective JGRPO(θ) = Eq,a, XG i=1 1oi X oi t=1 (Oi,t − βdi,t) XG i=1 where Oi,t = min ci,t(θ)Ai clip ci,t(θ), 1 −ϵ, 1 + ϵ di,t = DKL (πθ∥ πref) How the training signal is built The authors introduce confidence-weighted advantage to re-scale normalized gradients according to pseudo-label reliability Pseudo-label confidence is estimated via majority-vote consistency, Conf = 1/M PM m=1 1m=1 yˆ, and this is used to weight the normalized advantage: Ai = ri − mean(r1, r2,.,., G) std(r1, r2,.,., G) · f(Conf) The function used is f(Conf) = exp(Conf), which performs best overall This mechanism aligns the update magnitude with signal quality, as higher-confidence pseudo-labels receive larger weights than lower ones Where the gains come from The second key innovation is multiple-attempt sampling, which detects rollout collapse and sequentially selects the first non-identical group Algorithm 1 describes this sequential multi-attempt sampling strategy This strategy mitigates advantage collapse by selecting the first group with non-identical samples when earlier groups fail to be diverse The combination of confidence weighting and multiple-attempt sampling consistently improves performance with an average gain of 1.31% Empirical results Experimental results demonstrate that AQA-TTRL achieves an average improvement of 4.42% on Qwen2.5-Omni 7B and 11.04% on the 3B model across four benchmarks Notably, the adapted 3B model surpasses the static 7B model under direct inference on average (64.86 vs.

Rosa: 64.39) The results show that our method outperforms DIMV and SFT trained on the same pseudo-labels This demonstrates the potential of LALMs to self-improve under a label-free training paradigm at test time The authors justify their approach by demonstrating a strong positive correlation between model confidence and correctness on test data We propose AQA-TTRL, a label-free test-time adaptation framework for audio question answering By combining majority-vote pseudo-labeling with GRPO-based optimization, along with confidence weighting and multiple-attempt sampling, the framework enables stable self-improvement on unlabeled test data Experiments on MMAU (test-mini/test), MMAR, and MMSU demonstrate consistent gains Notably, the self-improved 3B model surpasses the direct inference performance of the unadapted 7B model.

Title and authors: Rosa: Alright, let's talk about the paper itself, "AQA-TTRL: Self-Adaptation in Audio Question Answering with Test-Time Reinforcement Learning". It’s a lot to unpack.

Dev: Yeah, it’s a framework that uses test-time reinforcement learning to allow large audio language models to adapt on the fly using only unlabeled test data.

Rosa: So, the key thing is that instead of needing expensive supervised fine-tuning, which takes lots of labeled data, this method lets the model evolve during testing just by generating pseudo-labels and optimizing through reinforcement learning.

Dev: It starts with majority voting over several model outputs to create a consensus pseudo-label and then uses that signal to guide the policy optimization process.

Rosa: And they’re not just using any RL method; they specifically adopt Group Relative Policy Optimization, or GRPO, which is a technique from their work on improving reasoning ability in language models.

Dev: So, it builds on existing ideas where reinforcement learning with verifiable rewards has been shown to improve LLMs without needing explicit reward models like DeepSeek Math or DeepSeek-R1-Zero.

Rosa: The authors are essentially taking those concepts and applying them directly to audio question answering by using RLVR, or Reinforcement Learning with Verifiable Rewards, at test time.

Dev: And the core of this paper is showing that this specific combination of LALMs and RLVR can outperform supervised fine-tuning on benchmarks like MMAU when using GRPO.

The paper's summary: Rosa: So, to summarize what they’re doing, they are taking LALMs, which are great at general audio understanding but get stuck after deployment.

Dev: And their goal is to solve that by proposing AQA-TTRL, a novel framework for audio understanding that allows on-the-fly evolution through test-time reinforcement learning using only unlabeled test data.

Rosa: They achieve this by first generating pseudo-labels via majority voting over multiple outputs, which acts as a self-generated reward signal.

Dev: That pseudo-label then guides the policy optimization using GRPO, and they introduce two specific innovations to handle noise: confidence weighting for the advantage and multiple-attempt sampling to prevent advantage collapse.

Rosa: So, it’s about creating a self-adaptation method that builds a closed loop for self-improvement using only test data to keep things stable.

Dev: It’s about using these mechanisms to ensure the pseudo-labels are reliable enough to guide the model without introducing too much noise into the training process.

The paper's improvements: Rosa: Now let's talk about how they actually make this work better than just having a basic setup. They introduce confidence-weighted advantage to re-scale those normalized gradients according to pseudo-label reliability, which means higher-confidence pseudo-labels get larger weights in the update calculation.

Dev: That’s smart because it aligns the update magnitude with signal quality, so the update size is proportional to how good that signal is.

Taro: And then they have multiple-attempt sampling, which detects rollout collapse and sequentially selects the first non-identical group when earlier groups fail to be diverse.

Rosa: This sampling strategy keeps the learning process moving by ensuring that high-confidence samples are generally of higher quality, so they contribute effectively to learning without stalling updates.

Dev: So you've got the confidence weighting handling one side of the noise, and this sampling strategy handling another side, which is really important for a stable training loop.

Conclusion: Rosa: To wrap up this discussion on AQA-TTRL: Self-Adaptation in Audio Question Answering with Test-Time Reinforcement Learning. The core takeaway is that they’ve shown how to enable stable self-improvement on unlabeled test data for LALMs by combining majority voting, GRPO optimization, and these two key innovations.

Dev: The results are substantial gains of four point four two percent on Qwen2 point 5-Omni 7B and eleven point zero four percent on the 3B model across MMAU, MMAR, and MMSU benchmarks when compared to static models trained only with those same pseudo-labels <ref:2510.05478#pg1,Qwen2.5-Omni 7B and 11.04>.

Taro: And that adapted 3B model actually surpasses the direct inference performance of the unadapted 7B model on average, which is a pretty striking result <ref:2510.05478#pg1>.

Rosa: So this framework shows that targeted on-the-fly adaptation can be a parameter efficient way to complement model scaling for these large audio models. It proves they can genuinely improve performance without needing massive amounts of labeled training data.

Dev: We're leaving the discussion on AQA-TTRL here, but we’re ready to look at what comes next in the research space.

More episodes

← Home