Beyond Text Following: Repairable Arbitration Reversals in Audio-Language Models
summary
The gist
Audio-language models (ALMs) often follow text that conflicts with audio, even when the audio evidence is clear The same-audio branch prefers the audio-supported answer…
In short
Audio-language models often fail when they follow text because they prefer text-supported answers even when clear audio evidence exists. The study found a 'repairable arbitration reversal' where models can support the audio answer, but joint input reverses the decision. A new decoding rule, GACL, fixes this by interpolating between joint and same-audio scores to improve accuracy without retraining.
Key concepts
- Repairable Arbitration Reversal
- This is a specific failure pattern where an AI model shows conflicting preferences: one branch (same-audio) favors the audio answer, while another (joint) favors the text answer. This reversal can be fixed by modifying how the model makes its final choice.
- Gated Audio Counterfactual Logit Correction (GACL)
- GACL is a training-free decoding rule designed to fix these arbitration errors. It uses branch disagreement and reference reliability to gate an interpolation between the joint and same-audio scores, allowing the model to correctly choose the audio answer when it's available.
- Diagnostic Behavior
- Researchers tested what happens when they deliberately remove conflicting text from an audio prompt while keeping the audio fixed. This diagnostic test revealed that many errors are not due to perceptual failure but rather a preference shift caused by the joint input structure.
Terminology used across episodes
This episode discusses
- Beyond Text Following: Repairable Arbitration Reversals in Audio-Language Models · Paper Radio
- Understanding intermediate layers using linear classifier probes
- When Audio-LLMs Don't Listen: A Cross-Linguistic Study of Modality Arbitration
- Qwen2-Audio Technical Report
- Kimi-Audio Technical Report
- How Contrastive Decoding Enhances Large Audio Language Models · Paper Radio
- Voxtral
- Simple synthetic data reduces sycophancy in large language models
- Qwen2.5-Omni Technical Report
- Qwen3-Omni Technical Report
- Evaluating and Steering Modality Preferences in Multimodal Large Language Model
The paper
Beyond Text Following: Repairable Arbitration Reversals in Audio-Language Models · Read on arXiv
Yichen Gao, Yiqun Zhang, Zijing Wang, Yujia Li, Heng Guo, Xi Wu, Xiaocui Yang, Shi Feng, Yifei Zhang
Northeastern University, China · Shanghai Artificial Intelligence Laboratory, China
DOI: 10.18653/2026.emnlp-main.1088
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Beyond Text Following".
Jane: Here's the gist of the paper. Audio-language models (ALMs) often follow text that conflicts with audio, even when the audio evidence is clear The same-audio branch prefers the audio-supported answer, whereas the joint branch prefers the text-supported answer This pattern suggests that the relevant audio evidence is encoded but loses in arbitration GACL improves nAUC by 17.8 points over the best contrastive baseline and transfers without retuning to vision–text arbitration (up to +40.5,pp) The core finding of the study is that a repairable arbitration reversal exists where models can support the audio answer, but the joint input reverses the decision This reversal is localized to an answer-position residual stream and its direction tracks output score differences at the rank level with Spearman ρ=0.93 The proposed GACL decoding rule interpolates between joint and same-audio scores, improving conflict audio-following accuracy by 17.8 nAUC points under a 5 pp faithfulness budget This method generalizes to vision–text arbitration without retuning, achieving up to +40.5 pp gain GACL's success is attributed to its diagnostic-derived gating and bounded interpolation mechanism Diagnostic Behavior of Arbitration Failure The study examines a same-audio counterfactual that keeps the audio fixed and removes only the conflicting text to measure preference shift Across five ALMs and four conflict tasks, 64.1% of conflict samples show a sign flip where the same-audio branch prefers the audio-supported answer, whereas the joint branch prefers the text-supported answer This pattern is referred to as repairable arbitration reversal Activation patching localizes this reversal to the answer-position residual stream, and patching effects closely track output candidate-score differences (Spearman ρ=0.93) The diagnostic region O is defined by MA > 0 and MJ < 0, indicating the audio answer is available in the same-audio reference branch but text-favored in the joint branch This behavioral evidence suggests that many conflict errors are not reducible to perceptual failure, as removing only conflicting text often recovers the audio-supported answer Mechanism of Repair and GACL Implementation The authors propose Gated Audio Counterfactual Logit Correction (GACL), a training-free decoding rule that interpolates between joint and same-audio reference scores gated by branch disagreement and reference reliability The GACL formula is defined as Nout(x) = 1
ˆyA ≠ ∅: · 1
ˆyA ≠ ˆyJ: , ∆A(x) = sA(ˆyA) − max y∈H(x) sA(c), RA(x) = clip(∆A(x)/τA, 0, 1), and α(x) = clipλ RA(x) Nout(x), 0, 1 The rank alignment between the hidden repair direction and the observable difference sA − sJ is Spearman ρ = 0.93 across twelve model–task configurations GACL's performance is robust, achieving a macro average of 0.81 rescue under a 5 pp faithfulness drop budget Performance and Generalization Under a strict 5 pp faithfulness-drop budget, GACL improves conflict audio-following accuracy by 17.8 nAUC points over the best contrastive decoding baseline Applied without retuning to vision–text arbitration on MC2, the same rule raises adversarial accuracy by up to +40.5 pp GACL dominates in the low-drop regime, showing gains of 51.0 nAUC for Qwen2-Audio-Inst.
Tom: on AQA at a 5 pp budget The method is sensitive to reference reliability, as GACL stays conservative when the audio reference is weak, defining an operating range where it converts repairable opportunities Ablation and Stress Testing Component-level ablation shows that the answer-side residual patching reaches a macro average of 0.
Paper summary: Jane: What they claim is that this preference shift happens in about sixty-four point one percent of their test cases, where you keep the audio fixed and remove only the conflicting text, and you see a sign flip.
Lu: That pattern suggests that the relevant audio evidence is encoded but it loses its influence once that conflicting text is added to the joint input.
Tom: They call this pattern repairable arbitration reversal, and they've localized this reversal to a specific part of the model called an answer-position residual stream.
Meng: So when you look at that stream, you can see exactly where the preference is shifting compared to just looking at the score differences between the audio branch and the joint branch.
Jane: And they found that this alignment tracks those score differences really well, with a Spearman correlation of zero point nine three across twelve different model and task setups.
Conclusion: Tom: So, to wrap up, this paper introduces Gated Audio Counterfactual Logit Correction or GACL, which is a training-free decoding rule.
Jane: It works by interpolating between the joint and same-audio reference scores, using a gating mechanism based on how much disagreement there is between the branches and how reliable that audio reference actually is.
Lu: And they've shown that under a five point five faithfulness drop budget, this GACL method improves conflict audio-following accuracy by seventeen point eight nAUC points over the best contrastive baseline.
Meng: The real kicker is that this rule transfers without any further training to vision and text arbitration, actually boosting adversarial accuracy by up to forty point five points when applied on a different task.
Tom: It sounds like they found a way to fix this specific kind of conflict without having to retrain the whole model, which is pretty important for deployment.
Jane: It’s about finding that specific repairable reversal and then building a simple decoding rule that leverages it.
More episodes
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback
- 2511.17264-Notes on Stack Machines and Quantum Stack Machines
- 2601.15318-On the closest balanced game